Possible caveats in mixing 32-bit and 64-bit App-V packages and environments

Ever since Microsoft introduced support for 64-bit environments in App-V, for both 64-bit applications (needed less often still, but will be important going forward) as well as for operating systems (more pressing need especially since Windows 7 which is the first truly usable 64-bit Windows client OS), it has been a stated fact that you generally should not mix App-V packages bitness-wise.
To that effect, Microsoft even has the supportability matrix stating that running 64-bit Sequencer generated packages aren’t supported on the 32-bit systems, at all. I couldn’t find the authorative source for this matrix, but Kevin’s old post about sequencing 64-bit applications does have it, for instance.

Despite this whole thing being “public knowledge” for anyone working with App-V packaging, I don’t think it really has been explained properly just why it might be a bad idea to do mix-and-match and so I thought of collecting various reasons here in this post. Please note that all of these hypothesis are based on my own research and conclusions, and there could be still more reasons or technical subtleties on the matter that I have not covered here.

Problem #1 : App-V variable mappings

As you likely know of, both App-V Sequencer and Client contain a functionality for mapping well-known items, such as Windows’ well-known directory paths, user’s name, SID and so forth in to a “neutral” variable form. In App-V terminology these are called Parse Items because, well, they parse from one form into another. I like to call them intermediate values since in actuality they are used to represent something after transformation/translation (or encoding) and before it’s transformed/translated (decoded) again back to a presently valid representation of whatever that is we translate between the systems and users. For the sake of this discussion, let’s use the term “variable mappings”.

Variable mappings are a static property in App-V because you cannot add your own. There was possibility for this in Sequencer during SoftGrid times, but it served no functional purpose and so it was removed during SoftGrid 4.X series (maybe even in 4.0 which was the “collateral damage” version in that it was long delayed due to MS acquisition). Since mappings are a static property, both Sequencer and Client has the same list built-in to them. You can actually observe this list in the Sequencer’s Options menu on a Parse Items -tab:

(don’t you just love the fact that the UI still has big, honking, blank space between the list and buttons as a reminder where additional buttons were for adding/editing/removing your own mappings)

As you can see from the screenshot, there’s Parse From column and Parse To column plus mapping type for each row. Each row is a separate mapping entry.
All Parse To entries – or the actual mapping variables – look like a environment variables in that they have percentage -sign as first and last character, but in reality most of the comes from something Microsoft used to call constant special item ID list and what application developers use/used to use when referring to well-known folder locations in the system (hey, sounds familiar purpose doesn’t it?!); they are deprecated now in Windows and KNOWNFOLDERIDs should be used instead. But we in App-V land still use these! Note that some of these mapping variables, on the other hand, are completely unique to App-V and do not have equivalent concept in Windows API programming.

The underlying idea of all of this is that during post installation monitoring, during scanning of the changes in the environment, when Sequencer sees some text in the registry key name, value name or value data that is also one of the entries in the From –column it will change that piece of text into equivalent text (variable) in the To –column.
If there are multiple different matching values in the same text, then they all are converted into matching variables, except in the key names where only the first variable-eligible instance of text for the whole key path is actually changed; rest of the text remains as-is. To see what I mean by this, try to create a new, empty, key under HKCU and naming your key per your account’s SID string under installation monitoring, then observe the virtual registry in Sequencer’s UI and see that you don’t have two %SFT_SID%s there in the nested path.. this has to be some sort of technical oversight or limitation in the VREG implementation (which probably dates back to very first SoftGrid version).

To illustrate with an actual example how the mapping works: if one of my registry values in some registry key were named after “C:\Program Files\Microsoft Office” it would be changed into “%CSIDL_PROGRAM_FILES%\Microsoft Office” and said registry value would internally be flagged as having this variable within (that’s why using variable names manually in Sequencer’s Virtual Registry UI does not work as expected, there is nothing to tell to the Sequencer that variable you entered manually actually needs any “special attention”).

In addition to virtual registry, this same mapping process also applies to directories that are created in the VFS part of the App-V package’s internal directory structure i.e. directories that were modified or created over the real partitions. Under VFS, the purpose is not to change directory or file names directly (as they appear inside the package) but modify paths into which these filesa or directories map to (i.e. are virtualized as). 

And as you can probably already guess from the data in Mapping Type column, that little piece of information drives the decision process so that all variable mapping entries with VRG in it are taken into account when Sequencer processes the registry data and VFS –marked entries are considered when VFS folders are encoded. Mostly every one of the entries in the list are both VRG&VFS, but few odd ones are only VRG or VFS because changing text to variable form based on those entries would have no meaningful purpose on the other one. Like the one in the screenshot –  WIN7X32 – which is the name of the local computer and thus would make no sense to start touching on any of the VFS’ed directories having this text as part of its path.

So far so good; now we know that the Sequencer changes some of the texts in virtual registry and in virtual filesystem directory paths into variable form. And the process is only for these entities: contents of any of the files in the package are not looked into by the Sequencer and modified, so INI files, XML files and such don’t get the special love from App-V which is actually still another potential bitness conflict point!).
When this package is then executed at the client under App-V Client’s virtual environment, it does the process in reverse and maps variables back into currently valid unencoded texts. So my “%CSIDL_PROGRAM_FILES%\Microsoft Office” becomes again “C:\Program Files\Microsoft Office” or “D:\Progam Files\Microsoft Office” if the system drive is not C: … or does it really?

The answer is that is does not necessarily work this way, if the package was created in different bitness system! And herein lies the dragons for mixed bitness usage..

In order to add x64 support in App-V, one of the things Microsoft had to do was to introduce whole bunch of new variables in 4.6 in addition to existing ones. This is because previously App-V only knew about well-known folders as they exist on 32-bit systems; Program Files, System32, Common Files, Application Data and so forth. But in the 64-bit Windows there are some new well-known folders, namely the following ones:

• Program Files (x86)
• Syswow64
• Program Files (x86)\Common Files

In addition to these folders, Microsoft also implemented additional weird variable-to-variable mappings such as %commonprogramfiles% and variables for previously missing standard folders like “C:\Windows\system32\catroot” and “C:\Windows\system32\spool” (and few others still). You can see these in the 4.6+ Sequencer’s Parse Items –list (almost all new variables start with %SFT_):

To make things as compatible with existing 32-bit packages from previous versions as possible, Microsoft did not invent new variable names for 32-bit directories on 64-bit Windows, but rather re-used the previous mappings as 32-bit versions of the directories and came up with new names for “native” directories, so that following is true on 64-bit system:

• %CSIDL_PROGRAM_FILES% = C:\Program Files (x86)
• %CSIDL_PROGRAM_FILES_COMMON% = C:\Program Files (x86)\Common Files
• %CSIDL_SYSTEM% = C:\Windows\Syswow64
• %SFT_PROGRAM_FILES_ X64% = C:\Program Files
• %SFT_PROGRAM_FILES_COMMON_X64% = C:\Program Files\Common Files
• %SFT_SYSTEM32_X64% = C:\Windows\System32
• …and similar entries for short-name versions of these directories, otherwise the same but with ~% as an ending instead of %

…but on the 32-bit Windows these are mapped like this:

• %CSIDL_PROGRAM_FILES% = C:\Program Files
• %CSIDL_PROGRAM_FILES_COMMON% = C:\Program Files\Common Files
• %CSIDL_SYSTEM% = C:\Windows\System32
• %SFT_PROGRAM_FILES_ X64% = C:\Program Files
• %SFT_PROGRAM_FILES_COMMON_X64% = C:\Program Files\Common Files
• %SFT_SYSTEM32_X64% = C:\Windows\System32
• …and similar entries for short-name versions of these directories, otherwise the same but with ~% as an ending instead of %

Assuming that one registry key example from above, it would decode differently when the package is run on 32-bit system or 64-bit system; a potential for messing things up. Granted, if application refers to a directory where it installed its own resources then the VFS would in most cases take care of placing the directory in correct location mapping-wise. But if instead application happens to capture any paths to components not in the package (pre-requisites already installed on the base or Windows components), they could point to a “wrong” Program Files, or “wrong” Common Files or “wrong” System32 structure.

A variation of the same issue happens when same key contains values which decode differently on 64-bit system but has conflict on 32-bit system; this actually happened to me during inspection of 64-bit originated Office 2010 package (I think it was O2010, but some version of Office anyways) which had one virtual registry key in it that contains several registry values each with folder path as a value name.
Unfortunately amongst of those value names were (paraphrasing here) “%CSIDL_PROGRAM_FILES%\Microsoft Office 14” and “%SFT_PROGRAM_FILES_ X64%\Microsoft Office 14”! This means that when such a package is run on 32-bit system, both will decode to exact same name and I don’t think App-V Client’s virtual registry handling can cope with this (because Windows registry cannot have duplicate value or key names as siblings).

And it’s not just the registry, if you have both 32-bit Program Files and 64-bit Program Files in the VFS, there could be potential conflict there as they both decode to same physical path on the client when run at 32-bit system. Not to mention that it is very likely that in such case there could be genuinely 64-bit executables or DLLs in the package. And same goes true for other VFS folders having both 64-bit and 32-bit versions in the package.

Hey, that’s still not all of it! The OSD file can also have variable mappings in it, since they are supported at least in CODEBASE FILENAME and in WORKINGDIR elements. That’s the third place where mappings can make unintentional mess of directory references.

Problem #2 : Virtual registry is EXACTLY as it is, except when it isn’t

Now, having seen the potential conflicts in how App-V encodes and decodes the variables (ironically meant to make packages as portable as possible), let’s look at the second possible problem for trying to run fully 32-bit packages or packages having 32-bit parts and created with a 64-bit Sequencer on a 32-bit Client: how App-V mounts the virtual registry.

(Note that truly 64-bit packages i.e. ones having native 64-bit applications inside are left outside of this discussion for the simple reason of them not even being able to execute on 32-bit Windows)

Like you have undoubtedly witnessed during your App-V virtualization packaging, virtual registry is displayed and stored in the package in the manner of having direct one-to-one relationship with what you see in Virtual Registry tab in Sequencer and how entries appear in the client. Yes, App-V likes to call HKLM as “MACHINE” and HKU as “USER”, but that’s just for display purposes. What’s under MACHINE in virtual registry is what gets virtualized under HKLM on a client and what’s under USER gets virtualized under HKU (usually %SFT_SID% which resolves to user’s SID which is the same as what Windows shows as non-physical hive of HKCU).

But, on a 64-bit Windows you have the concept of Wow6432Node key, which lives under HKLM\Software, HKCU\Software and under Classes –node under both of those. This is the registry-equivalent of mechanism in file-system discussed earlier during variable mappings: registry writes and reads for all 32-bit executables are redirected (unless they really make the effort of avoiding this using specific Windows API calls) silently under Wow6432Node branches and if looking at registry from 32-bit application (like 32-bit regedit.exe which is in c:\Windows\Syswow64), they don’t in fact see any of this happening but see the contents of Wow6432Node directly under HKLM\Software, for instance.

Exactly how this, then, relates to App-V and virtual registry?

To illustrate the point, when looking at the virtual registry of 32-bit version of Autodesk Design Review but sequenced on 64-bit Sequencer (using AVE here, but the same can be done with the Sequencer itself) you can see that some of the entries are directly under Software but some of the entries are under Wow6432Node:

Looking at the same application, but sequenced on 32-bit system we can see that registry now looks “normal” without any Wow6432Node’s in-between of Software and our main application’s registry settings:

These differences happen because sequencing 32-bit application on 64-bit system causes all registry entries for 32-bit application (during install time) to be redirected, to ensure that 32-bit application will see the registry as it should when it runs. But when sequencing on 32-bit system, there isn’t any redirection taking place or needed, so registry keys are where they are supposed to be.

For our 64-bit App-V package on 32-bit client mixing scenario, the problem begins here; let’s first see how our 64-bit package looks on 32-bit client when we start registry editor into the virtual environment. Remember, this is still 32-bit application, only having been packaged on 64-bit system:

Hmm, that’s interesting: there isn’t any Autodesk key under Software as we would assume to be the case in 32-bit Windows, but instead extraneous Wow6432Node appears which contains ONLY those registry keys we had in the package…

Next, let’s have a similar registry look at how 32-bit originated package is seen on 64-bit Windows by using 32-bit regedit.exe:

There is the Autodesk -key in the expected location, which is what we wanted. And no indication of Wow6432Node under Software, so this would be the view for the actual 32-bit applications too which see the Wow6432Node -key as Software -key.

For comparison, let’s look at the virtual environment using 64-bit regedit.exe:

Well, interestingly there are now actually two sets of Autodesk keys, both in 64-bit portion of the registry and 32-bit portion of the registry! It seems that when you have entries in the “normal” portion of the registry (i.e. not under Wow6432Node in its various possible places), coming from 32-bit Sequencer, then they all will be virtualized over both 64-bit and 32-bit locations in the registry and available for both 32-bit and 64-bit applications.

So how exactly App-V knows to do all this? Why didn’t it assume Autodesk key to be real 64-bit registry key and just overlay registry like 32-bit App-V Client did for our package coming from 64-bit Sequencer? Or why, on the other hand, it does not do any redirection for actual 64-bit Sequencer generated registry?

The answer lies in the fact that App-V actually stores indication in the virtual environment configuration file (osguard.cp) if the package is “32-bit” or “64-bit”. If this flag tells that the package contains 32-bit entries, then the 64-bit Client knows to do additional Wow64 redirection (or copying) to all keys under HKLM\Software and those various other places. If the flag tells that the package is 64-bit, then redirection is disabled. 32-bit App-V Client does not seem read this flag at all or just ignores it (which is understandable) so there is no redirection going on back from Wow6432Node to the base key in that scenario.
And if want to verify if your package is coming from 32-bit source on a 64-bit App-V Client, you can actually see when redirection is triggered by peeking at client’s log file (sftlog.txt). There is no indication whatsoever on App-V package’s internal bitness in any other place:

(btw, those error lines beneath our WOW64 redirection message seem to be mostly harmless and happening sometimes with 4.6)

While everything works correctly under 64-bit Windows with App-V doing extra heavy lifting with redirections, one thing is certain from the tests. And it is the fundamental problem for running 64-bit Sequencer generated packages on 32-bit system: Copying of the keys does not happen from Wow6432Node to base key and this means that 32-bit applications won’t see any of their entries if you have sequenced it on 64-bit system and try to deploy to 32-bit Windows. App-V just overlays the virtual registry over the physical one without any consideration of mixed usage, whereas 64-bit App-V Client does make sure that your existing 32-bit packages will work correctly registry-wise when executed on a 64-bit system.

Conclusions

To summarize all the things discussed in this article, there exists at least two different set of bitness –related issues which are [some of] the reasons for Microsoft not supporting 64-bit system created packages on 32-bit client systems at all, and cautioning against running it even other way around.
Under good conditions, the latter case usually works automagically, but there still is possibility for some packages to break in subtle ways which are problematic to diagnose. But, I’m sure they have done their best during version 4.6 development to prevent that from happening, since all the existing packages were 32-bit up until 4.6’s original release.
In the case of doing things fully unsupported way (the first case); applications that do not rely on registry settings at all may work correctly, but registry-heavy applications most likely do not function since they are not able to see any of the virtualized registry entries. Add COM registrations etc. into the mix, and you easily have hundreds of registry keys and values to check for!