Is wp co_mextract_feb13

1. WHITEPAPER
What Installation Authors Need to Know
about COM Extraction
By Josh Stechnij
Senior Software Engineer, Flexera Software

2. What Installation Authors Need to Know about COM Extraction
2
What Installation Authors Need to
Know about COM Extraction
COM extraction was originally created to be a tool for
populating Windows Installer COM/registry table data
without having to manually determine and populate data by
hand from a given COM server. Windows Installer provides
the Class, Registry, Typelib (deprecated), and ProgId tables
to automatically install COM server registration data in a
way that allows for consistent rollback on failed installations
and also provides a way for Windows Installer to hook into
application usage to verify installation consistency (install
on demand/self-resiliency). Over the years since COM
extraction was first provided, InstallShield has updated
COM extraction functionality in an attempt to provide more
accurate extraction and useful functionality. This white
paper aims to provide some background on how COM
extraction works in InstallShield and some challenges that
have come up when attempting to use COM extraction.
COM Extraction Methods Employed in InstallShield
In order to extract COM information from a COM server,
some method needs to be used to capture COM data
from the file. While static analysis of the file (type library
and registration scripts) could be used, this is limited in
functionality since it can only capture declarative data that
is contained in obvious places in the file. More flexible
methods could be used through dynamic capture of
registration data as it is written to the machine while a file is
being registered. Since the introduction of COM extraction
in InstallShield, a few different dynamic methods have been
used in an attempt to provide the most accurate extraction.
API Hooking
The original support for COM extraction was implemented
through the use of API hooks. When COM extraction was
started for a given file, APIs in Kernel32 and Advapi32
were hooked by COM extraction to monitor registry
modifications on the system. API hooking provided a
relatively clean approach to capturing registry changes such
that the system was affected minimally by the registration
process. One downside to this approach was that the
registration itself was not insulated from the system. If a
given file was already registered on the machine prior
to running COM extraction, this would alter how most
COM registration proceeded (no changes to the system
would typically occur since the registration took note that
its information was already present on the machine). To
work around this issue, the file to be registered is first
unregistered. Then, the capture through API hooking is
enabled and the file registration is started.
Capture of in-process COM servers such as DLLs, OCXs,
etc. is a fairly simple process of hooking APIs in the
COM extraction process. However, capturing registration
information from out-of-process servers in the form of
EXEs requires the use of a helper DLL that is injected into
the process to capture. At the time this method of COM
extraction was created, injecting DLLs into an arbitrary
process was rather simple. Over time, with the introduction
of more heavily secured operating systems such as
Windows XP SP2, Windows 2003 Server, and Windows
Vista, DLL injection became increasing difficult and less
reliable. In addition, API hooking also became less reliable
after COM servers started building with Visual Studio 2005
(likely due to changes in the registration implementations).

3. What Installation Authors Need to Know about COM Extraction
3
Registry Redirection
As API hooking became less reliable with changes in
OS and development tools, a new way of capturing
COM information was needed. Starting with Windows
2000, the operating system allows for redirection of all
registry accesses to private keys through the use of the
RegOverridePredefKey API. This API is expressly provided
for the purpose of software installations to allow for
capturing system registry changes.
Starting with InstallShield 12 (which provided mainstream
support for Windows Vista), COM extraction defaulted to
using registry redirection as a means of capturing COM
registration data during COM extraction. This method
provided a way to present a “clean” view of a system to
the file being registered, eliminating the need to unregister
the file prior to registering. This method resolved most
outstanding issues occurring with API hooking that had
cropped up over the years. In addition, this method has
the advantage that the machine state prior to and after
capturing registry information will not change. As with
most solutions, though, some tradeoffs appeared. Since
registry redirection presents a view of the registry that
is available in the redirected keys (which tend to be
completely empty), COM servers that depended on
existing registry information would no longer capture
correctly. Some workarounds to this were available, such
as writing dummy ATL registry keys needed to register
some components dependent on ATL. However, no generic
solution to missing registry dependencies is available, and
components that have such dependencies will not correctly
have all COM information extracted.
Kernel Mode Registry Filtering
In order to resolve the issues brought about by registry
redirection COM extraction while maintaining a similar
level of system isolation, and to provide some additional
support of .NET assemblies, InstallShield 2012 introduced a
new method of extracting COM information. On Windows
Vista and newer versions of Windows, InstallShield will use
kernel mode registry filtering to capture registry data for
COM extraction.
Kernel mode registry filtering is a Windows mechanism for
intercepting system-wide registry accesses through a kernel
mode driver. This support was first provided on a limited
basis with Windows XP. Windows Vista improved registry
filtering such that it was generally usable to capture registry
information from any process running on a system. Registry
filtering is used by utilities like Sysinternals’ Process Monitor
to able to capture registry accesses across processes on
a system.
Through registry filtering, the COM extraction provided
in InstallShield 2012 and newer versions results in a
hybrid approach that combines aspects of the API hook
and registry redirection methods. A COM server being
registered with register filtering COM extraction will have
a view of the actual system registry like API hooking. The
captured data is completely redirected by the filter driver
to a private InstallShield registry key, similar to the registry
redirection method. The end result is no DLL injection is
required and a registering component has complete access
to any registry dependencies it may have.
An additional benefit of registry filtering is being able to
capture registry modifications from any process, not just a
process that is registering COM data. This now allows for
capturing registry changes for .NET COM interop made
by regasm. Since creating a registry file with regasm
does not include code-level registry changes made by an
assembly, the registry filtering can now be used to capture
registry changes made by registering an assembly for COM
interop with regasm. In addition, C++/CLI assemblies that
“register” can now also be captured correctly (previous
COM extraction methods resulted in failures in the assembly
or CLR that caused the extraction to fail).
Specifying Exclusions for COM Extraction
To prevent InstallShield from extracting undesired COM
data using any of the available extraction methods (either
at build or design time), the Filters.xml file contains a list of
registry keys that need to be excluded. Filters.xml can be
customized as needed. It is available in the Support folder
of an installed copy of InstallShield (typically located in C:
Program FilesInstallShield<Version>).
The <Exclude> element in the Filters.xml file contains a
subelement for each registry key that the COM extraction
process should ignore. By default, the <Exclude> element
has subelements for common system registry keys that
are required.
Following is a sample of a registry subelement that blocks
changes to an InprocServer32 registry key, all of its values,
and all of its subkeys:
<Registry key="HKEY_LOCAL_MACHINE
SOFTWAREClassesCLSID
{00000231-0000-0010-8000-00AA006D2EA4}
InprocServer32"/>
Following is a sample of a registry subelement that blocks
changes to only the default value of an InprocServer32
registry key:
<Registry key="HKEY_LOCAL_
MACHINESOFTWAREClassesCLSID
{00000231-0000-0010-8000-00AA006D2EA4}
InprocServer32" value=""/>
Following is a sample of a registry subelement that blocks
changes to only the ThreadingModel value name for an
InprocServer32 registry key:
<Registry key="HKEY_LOCAL_MACHINE
SOFTWAREClassesCLSID
{00000231-0000-0010-8000-00AA006D2EA4}
InprocServer32" value="ThreadingModel"/>

4. What Installation Authors Need to Know about COM Extraction
4
The following table describes the recognized attributes for
the <Registry> subelement in the Filters.xml file.
Any other attributes for the <Registry> subelement are
optional and are not recognized by the COM extraction
process. This enables additional attributes to be used as
internal comments to identify why an item is being excluded.
How InstallShield Populates the Windows Installer Tables
with COM Extraction Data
The following table identifies how InstallShield maps the
COM extraction registry entries to Windows Installer
table data.
The data in the Windows Installer tables allows for rollback
of the COM data in the AppId, Class, MIME, ProgId, and
Registry tables.
Troubleshooting COM Extraction Issues
With each method of COM extraction, some issues can
occur with specific COM servers. The issues typically
stem more from files that do not register in a normal/stock
manner. The best bet in troubleshooting issues with COM
extraction is to use a tool named RegSpyUI. This tool runs
COM extraction outside of the InstallShield IDE and can
be set to not filter any captured registry data. RegSpyUI
is available in the Support folder of an installed copy
of InstallShield (typically located in C:Program Files
InstallShield<Version>). This tool can be used to perform
one-off testing of files that do not seem to be extracting
correctly in the InstallShield IDE.
Another available method of troubleshooting is to change
the COM extraction method that InstallShield uses. All three
methods of COM extraction are still available in recent
versions of InstallShield. The method used can be changed
by setting the value of the UseAPIRegistryHooks registry
value located in HKEY_LOCAL_MACHINESoftware
InstallShieldRegSpy (on 32-bit machines) or HKEY_LOCAL_
MACHINESOFTWAREWow6432NodeInstallShield
RegSpy (on 64-bit machines). The following values are
available:
Attribute Description
key This attribute must be lowercase. The value of this
attribute (for example, HKEY_LOCAL_MACHINE
SOFTWAREClassesCLSID00000231-0000-
0010-8000-00AA006D2EA4}InprocServer32
in the above examples) indicates the name of the
registry key that needs to be filtered.
value This attribute is optional:
• To block changes to an entire registry key, do
not include the value attribute.
• To block changes to the default value of the
specified key, set the value of this attribute to
a null value.
• To block changes to the name of a value
of the specified key, set the value of this
attribute to the name of that registry value.
Data Found During COM
Extraction
Equivalent Windows Installer
Data
AppID entries in HKEY_
CLASSES_ROOT
AppId table
CLSID entries in HKEY_
CLASSES_ROOT
Class table
MIME entries in HKEY_
CLASSES_ROOT
MIME table, with a CLSID
association
ProgID entries that are
referenced by CLSID entries in
HKEY_CLASSES_ROOT
ProgId table
Miscellaneous registry data Registry table
InstallShield replaces file
references with a formatted
file key in short name format
([!filekey]).
InstallShield replaces
directory paths that are tied
to the component with a
component key path reference
([$componentname]).
UseAPIRegistryHooks value Value Description
Value not present Defaults to kernel mode registry
filter with InstallShield 2012 and
newer on Vista and newer
versions of Windows; defaults
to registry redirection on older
versions and platforms
0 API hooking
1 Registry redirection
2 Kernel mode registry filter with
InstallShield 2012 or newer
on Vista or newer versions of
Windows; invalid otherwise

5. What Installation Authors Need to Know about COM Extraction
5
By changing the COM extraction method it may be
possible that another method is able to successfully capture
COM registration information from a server that does not
work well with another method.
A Word About Seemingly Hung COM Extraction
Processes
A few reports of COM extraction hanging during build
(IsRegSpy.exe does not exit) have come from InstallShield
product support services. In nearly all of these cases,
registering manually with regsvr32.exe resulted in a
hung regsvr32 process. Upon further debugging of either
IsRegSpy.exe or regsvr32.exe processes, the offending DLL
was stuck in its DllMain function due to calling into another
DLL or otherwise violating the system loader lock hierarchy.
When authoring any in-process code (DLL, OCX, etc.) that is
loaded through LoadLibrary, it is absolutely critical that all
rules regarding what is safe or not safe to do in DllMain is
respected. Issues with DllMain resulting in deadlocks are a
fairly well known issue. If a DLL seems to be hanging when
performing COM extraction, check to see if it also hangs
with regsvr32.exe on the same or a similarly configured
machine. If it does, attach a debugger to see if the DLL is
stuck in its DllMain. Properly authoring a DllMain will avoid
this type of behavior.
Summary
This white paper discusses the various types of COM
extraction that are available in InstallShield and explains
how to switch from one method to another. This information
should help installation authors make appropriate choices
for COM servers as they design installations in InstallShield.

6. Flexera Software LLC
1000 East Woodfield Road,
Suite 400
Schaumburg, IL 60173 USA
Schaumburg
(Global Headquarters):
+1 800-809-5659
United Kingdom (Europe,
Middle East Headquarters):
+44 870-871-1111
+44 870-873-6300
Australia (Asia,
Pacific Headquarters):
+61 3-9895-2000
For more office locations visit:
www.flexerasoftware.com
Copyright © 2013 Flexera Software LLC. All other brand and product names mentioned herein may be the trademarks and registered trademarks of their respective owners.
IS_WP_COMextract_Feb13