Configure Application Virtualization Explorer to run inside App-V 5.0 package

One of the support cases that we have had, since the release of Application Virtualization Explorer version 3.0 that started supporting App-V 5.0 packages, is how to run AVE itself inside 5.0 package. Sounds simple enough, doesn’t it: just add the files into .APPV file and go, right? Well, not so in this case…

Frustratingly enough, if you run the AVE’s binary that handles App-V 5.0 packages (AVE.exe) it does load without issues but once you try to open .APPV package the loading ends with generic error message (“Internal error during package load”):

The “classic” version of AVE that handles pre-5.0 packages does not exhibit this same issue and can operate just fine inside App-V 5.0 “bubble” (our Professional edition ships with both versions, Standard edition only with the new 5.0 capable AVE). So far, we haven’t been able to figure out how to fix this issue, even though we know exactly what’s causing it. Fortunately, due to an unrelated blog post by Dan Gough the workaround was finally found that makes it possible to work around the problem!

Technical background on the issue

When we have investigated the issue, after receiving support cases about it, we were able to trace the problem to the way AVE loads virtual registry. Or more precisely put, the way our underlying Lib-V handles the virtual registry loading (and assuming the symmetry of the operations, save of the virtual registry as well but we never got to that point). Lib-V is responsible for the actual heavy-lifting of App-V package handling (decoding, encoding, the programmatic object model), whereas AVE acts as a graphical shell to it and so the AVE itself does not do any work when loading the package.

As you may know, when Microsoft re-imagined the whole virtual application fileformat (.APPV) when they moved from 4.6 to 5.0, it was based on more standardized container format for Windows 8’s Metro/Modern/Universal apps, APPX, which itself was based on originally Office 2007’s new OPC format. While in the old SFT format the virtual registry was encoded in the binary file as custom data structure and thus Lib-V had to implement the parsing on its own, in the new .APPV container model the virtual registry is actually stored as regular Windows registry export file, named registry.dat. This was surely to make it more simple and not having to re-implement the wheel at the App-V level.

For us, this meant that instead of yet again hand-crafting a decoding/encoding code for what was standard Windows file-format (word “standard” here is bit stretch, as it’s undocumented by MS), we could perhaps use available Windows APIs to handle this file-format. If registry editor can save and load hives in this format, there surely has to be an API for it in Windows as well?

Yes, there is. But here’s the catch: up until Windows Vista the only API function for doing this operation is RegLoadKey() which, as unfortunate as it is, require SE_RESTORE_NAME and SE_BACKUP_NAME privileges for the code that calls it. Concretely, this means that unless you have specifically configured your account to have such privileges, the usual way to have them enables is to run with administrator -level account. Only other option starting with Vista would have been RegLoadAppKey() function that does operate under normal user account rights, but based on the testing it’s very inflexible and unreliable for our purposes so the older API was only option available to be used.

When it comes to saving the package, the operation has to be done on reverse, i.e. we constrcut the virtual registry inside the real registry and then just export it the registry.dat file using the RegSaveKey/RegSaveKeyEx API. Which, as you can now imagine, also has requirement of having SE_BACKUP_NAME privilege enabled. So, that’s what we had to use in Lib-V when doing App-V 5.0 -supporting code, and that my friends, is also why 5.0 -enabled AVE requires administrator -rights to run…

What we did not know at the time and what was found out later on when troubleshooting this issue of loading App-V 5.0 packages inside the App-V 5.0 virtual environment, is that for what ever reason calling the RegLoadKey API fails. I suspect that this is some limitation of how Microsoft has implemented virtual registry handling in App-V 5.0 Client. So no matter what we do from our code point-of-view, the package loading always stops when we try to mount the virtual registry file into the real registry if the code is executing under App-V virtual environment. And since the virtual registry cannot be loaded, the whole package loading is naturally stopped at that point.

For old legacy App-V packages this particular roadblock is not an issue with Lib-V/AVE, as we do all the parsing of the virtual registry and also serialization without any help from any Windows APIs. Tells something about relying on external factors such as outside APIs, isn’t it?

Workaround is found

As said, there is nothing we can actually do internal to the AVE (or Lib-V) that would help get around of the limitation. But, in the Dan’s blog post he mentions fixing virtualization issue with some in-house application using a feature that is part of App-V 5.0 packages: disabling the whole virtual registry subsystem. I suspect that in Dan’s case the actual reason why the in-house app couldn’t connect to DB with virtual registry enables might be related to the issue we’re having i.e. there’s some API function that that application calls that fails while inside the virtualized registry.

This method of flat-out disabling the whole subsystem is something we have not been thinking as a workaround solution for our problem, as we were only focusing on how to make package loading work if run inside the virtual registry. But if the virtual registry is disabled completely for the package, then the calls we make should pass unaltered to the real registry, right?

Turns out, this is exactly what happens!

If you go ahead, and turn the virtual registry subsystem for package having AVE inside it, the loading goes through just fine as does the save as well. With the AVE, you can achieve this from the Integration Subsystems/Virtual registry -node for the package:

By default, for all configuration files the subsystem for virtual registry (like others) is enabled. You can click the “Disable subsystem” link to turn it off for all dynamic configuration files. It’s unfortunate, but you cannot do the virtual registry subsystem disable directly inside .APPV file but have to use dynamic configuration file for this. Which means that when deploying the package, either deployment config or user config has to be used. Otherwise the virtual registry will be enabled.

After changing the state, the screen should read that the subsystem are currently disabled:

Save the package, and now you should have a working AVE inside the App-V 5.0 package as well!

Note that the obvious downside to the workaround is that you cannot pre-configure anything inside the AVE package which uses the registry, as any captured registry entries would not be visible to AVE. This includes all the settings you could change in AVE’s Options etc.

If you want to do the registry subsystem disable by hand without AVE, you can open up the configuration XML files into text editor and find the node (under <Subsystems>…</Subsystems> that reads <Registry Enabled=”true” /> and change it to <Registry Enabled=”false” />.

In closing, I would like to thank Dan once more for indirectly giving a solution to our long-standing issue. It sure helps to think things outside the box, once in a while ;-)