What everyone and their mother should know about VFS in App-V, pt. 2 – Client –based operations for VFS folders and special behaviour of merged folders

In the previous article in the series, we discussed the different modes of Virtual File System (VFS) entries in a general level and exactly how application sees the files (and directories) stored in VFS –virtualized folders. This time over we are taking a little bit closer look at the client’s behavior when it comes creating new files or directories on these VFS-virtualized directories, as opposed to just reading in files that were carried over from the Sequencer.

We will also discuss some specific technical aspects of using merged VFS folders and what that means to your virtual application.

Where does the data live?

For as long as the VFS –virtualized folders and files live as defined inside the package, it’s pretty understandable how they are supposed to behave. But what if we need to change some of those files runtime and especially add new files or directories under any folder that is added to the VFS configuration?

The case of modifying pre-existing files (that is, content that your App-V package readily stores inside it) is pretty easy and simple one to describe and understand, at least in a concept level. When you change any virtual file – and this applies both files inside VFS as well as elsewhere inside the package’s internal directory structure – the change is never written to real disk partition but the change persisted inside virtual environment (VE) for that package.

But since the package cache drive is in actuality a read-only drive (i.e. your sftfs.fsd cache file, which is by default stored under C:\ProgramData\Microsoft\Application Virtualization Client\SoftGrid Client, and mounted as Q:\ for example), that persistence of file data does not happen directly against the package cache file, but rather inside the .PKG delta files.

Note that I mention files as in a plural, because there are actually minimum of four PKG files for each App-V package in the given system:

• UsrVol_sftfs_v1.pkg -file in your own roaming Application data directory (%appdata%\SoftGrid Client).This file contains all changes to the files marked with User Data –attribute (in Sequencer) which were done by and subsequently will be visible to the processes started under your user account.  Usually this means applications you as a user launch and use from the App-V package. 
• UsrVol_sftfs_v1.pkg –file in per-machine App-V cache (C:\ProgramData\Microsoft\Application Virtualization Client\SoftGrid Client\AppFS Storage).This file contains all changes to the files marked with User Data –attribute which were done by and subsequently will be visible to few selected Windows system processes (like winlogon.exe) having access to Q: drive (by App-V Client’s file system filter, normally blocking non-virtual processes from accessing it).
• GlblVol_sftfs_v1_user_SID.pkg -file in per-machine App-V cache (C:\ProgramData\Microsoft\Application Virtualization Client\SoftGrid Client\AppFS Storage).This file contains all changes to the files marked with Application Data –attribute which were done by and subsequently will be visible to processes started under user account that has the matching SID mentioned as part of the PKG file’s name (for example, I would have files called GlblVol_sftfs_v1_S-1-5-21-4084310487-3401208622-2622696841-1116.pkg as that’s my account’s SID). Normally, you should see at least one with your account’s SID in the directory, but on machines having multiple users there’s one for each account  that has used that particular virtual application
• GlblVol_sftfs_v1_S-1-5-20.pkg file in per-machine App-V cache (C:\ProgramData\Microsoft\Application Virtualization Client\SoftGrid Client\AppFS Storage).This file contains all changes to the files marked with Application Data –attribute which were done by and subsequently will be visible to processes running under system account in the virtual environment. These processes are commonly virtual services started from the package.

As an example, here’s screenshots from both my personal cache directory, as well as per-machine directory of cached settings for the same package (PuTTY):

We could look inside these cached settings files (using AVE, of course) and see what’s been cached, but since PuTTY does not really store anything interesting in the file system or VFS which is the topic of this post, I’ll leave the VFS examination of the PKG files as exercise for the reader :-)

Getting back to our topic of where does the data live for Virtual File System (VFS) entries; we now know that if any of the files that are already in the package when initially deployed to the client were to change (subject to security descriptors being enforced for package and how’s NTFS ACLs are laid out for those files), the change will be physically written to one of these PKG files. And when App-V Client after the fact sees any I/O access being targeted for such a file, the calling application will actually access changed or branched file from PKG file instead of the one stored in the package data cache. It doesn’t really matter to the application itself, since the action is in the background hidden from it and everything still looks like normal for the application in a sense that the file still seems to be at Q:\… –path or somewhere in the C:\… path if using VFS.

In a related note, this whole thing also means that if you include some file with your App-V package during sequencing, that file is “condemned” to live inside VE for all of eternity (or at least during the lifecycle of that particular virtual application). This point is especially important factor to consider when deploying [virtual] applications that has things like database files or other data which is volatile in nature, and which starts changing after deployment. Usually the problem happens with packages that have pre-staged application or user data in %CSIDL_APPDATA%; helpfully created by the original application installer during sequencing time.

If we do not take any additional steps, like script a copy of that data from the package to the real user’s profile directory pre-launch (i.e. store it somewhere else inside the package besides VFS and copy it back to original location on a client), or exclude it away during sequencing – hoping that application can re-create it during first launch – we effectively leave the data to live 100% inside VE on a client. The problem in this case would be that any repair or remove/import –operation against the package could accidently lead to a reset of our data that instead should live independent of the application package.

Now, this remark handily leads us back to another topic of VFS already mentioned at the beginning of this article, which is the question of what about new data against VFS?

Client branch behaviour of VFS override vs. merged files

When you add new files or directories to folders that are included in package’s VFS configuration, what will happen actually depends on your VFS virtualization level or mode. If you have a fully virtualized VFS folder, then everything extra that gets added to that folder will always live inside VE. That is, they will be written to the PKG file(s) and NOT to the real equivalent folders even if those folders would happen to exists on a client machine!

Like the previsouly noted issue with databases etc. coming from the package itself, the very fact that the fully virtualized VFS folders will keep all additional contents inside VE has pretty important ramifications for you as a packager: It means, that any profile –related folders or any other folders over at VFS and carried with the package (because, hey, it’s very convenient to add virtual folders to the package since you don’t have to worry about that the end user will have credentials to add those folders physically on a machine should it be needed, especially for legacy applications needing all-writable folders under C:\, for instance) will be closely tied with the life-time of the package itself.

If you have, as an example, an application that creates and uses a profile folder under roaming application data (Firefox/Mozilla …!), and you would want to add or pre-stage some files to it using handy VFS technology – such as templates or configuration files for app pre-configuration – knowing that the application under user’s context will start “filling it up” afterwards, you are somewhat in trouble as those files would not be accessible outside the application itself should they need to be. At least not without additional trickery like launching command prompt or Windows Explorer into your virtual environment’s bubble.

Like with the data that has been shipped with the package itself, a problem likely surfaces when you need to repair the application for troubleshooting purposes, or migrate to another packaging of the same application or newer one (e.g. from Firefox 7 package to Firefox 8 package, without upgrading the existing App-V package so that the VE will be kept the same). If all of your user data is stored solely inside the VE, new/another App-V package won’t see a thing from that data!

In such a scenario, alternative option could be to use merged VFS folders.

Because when you use merged -state VFS folder, all new files or directories added to that VFS virtualized folder will actually go to the equivalent local folder in the same path. Remember, merged folder merges the contents of local folder with the virtual version of the same like we saw in the previous article in the series, so there’s going to be a local directory as a backing store where in App-V Client can pass-through the I/O for the new file(s) or directory.

Of course, even in this case there exists at least a one potential problem, because from the application’s point-of-view the data in such a folder could in fact be a logical whole made up from individual pieces (i.e. files).
Let’s say that one or two of those pieces come from the initial package (i.e. SFT file) and the rest of the pieces come from post-deployment actions done by the user with the application. Now, in a given moment after user has started using the application, things are lining up nicely because both VFS virtualized files and user-created physical files can be found in the same [merged] directory. But what if we go and repair/reset the App-V package on that client? Now all the VFS virtualized files are brought back to their initial state, which could mean that there are no longer references to the local – physical – files in them!

Out of my hat I can make a contrived example using the oft-mentioned Firefox, which has the profile directory laid out so that there’s INI file that contains list of all individual Mozilla profiles inside that data directory structure:

If I were to deploy Firefox package so that there’s VFS –virtualized profiles.ini -file in the package, but passing through Profiles –directory in merged state (actually, all %CSIDL_APPDATA%\Mozilla, %CSIDL_APPDATA%\Mozilla\Firefox and %CSIDL_APPDATA%\Mozilla\Firefox\Profiles would need to be in merged state as fully virtualized VFS folder cannot contain merged VFS folder), creating a new Mozilla profile from the Firefox’s UI would change profiles.ini file’s contents. But in this case actual profile directory will go to a physical %appdata%\Mozilla\Firefox\Profiles -location, which would create a problem if I then do Repair –action against the Firefox package from App-V Client since the post-repair state of the profiles.ini won’t list my additional profile anymore.

But if your application does not store things in this way, or you use VFS folders in consistent manner, then the model could work just okay using merged folders. And fortunately most of the time when sequencing normal applications, you just don’t have to think about these sort of issues as things go pretty automatic for VFS related folders.
It’s only when you have an application where you need to think how the data will live, only then you need to understand the fundamental working difference for overridden and merged VFS folders.

But wait, that’s not all for merged folders!

In a perfect world, the discussion about overridden vs. merged folder would end here. But, alas, we are not in the perfect world so there’s something more you need to know about merged VFS folders in particular.

Due to way App-V internally stores VFS folders, in the osguard.cp file inside your every SFT file, there’s specific technical limitation of how merged folder can appear in the VFS configuration. To best illustrate this limitation, please consider an App-V package with the following VFS configuration inside it:

That’s two subdirectories for 32-bit Program Files, one marked as merged and another marked as override. Both contain one virtual file. I named each with different names, so that we can see what’s stored low-level. After processing the resulting SFT file (and osguard.cp in it) with a tool that dumps each CP file section, allowing us to examine one that has the VFS mappings in it (Sec 0, if you are interested), we can see how those VFS mapping are stored by the App-V technology.

First mapping listed is the one for the fully virtualized Test Application 2 directory:

There’s both the source as well as the target paths for the mapping i.e. “\??\%SFT_MNT%\Merged VFS test\VFS\CSIDL_PROGRAM_FILES\Test Application 2\” from the package mapped to “\??\%CSIDL_PROGRAM_FILES%\Test Application 2\” -path. As we can see from the mapping, there’s explicit mapping stored for that fully virtualized directory.

Next mapping stored is the file under previously defined directory:

This mapping has again the full path both from the package’s mounted directory point-of-view as well as the full path where to it is mapped (%CSIDL_PROGRAM_FILES%\Test Application 2\FILE2).

Looking ahead in the Sec 0, we will see – not a directory, but – a file again:

This is the virtual file (FILE1) that we stored under Test Application –directory, which was the one directory left in merged state. As we can see, the file itself is mentioned in the VFS mapping configuration but there is not a sight of the “Test Application” directory.

Why is that?

Turns out that the reason is that App-V does not store merged folders on their own in the package file, at all! All mapping entries in the osguard.cp’s Section 0 refers to folders and files that are to be virtualized fully (which is the only option for virtual files), and merged folders are not stored directly but rather figured out indirectly from the virtual files that do not have any containing directory stored with explicit VFS entry.
What this amounts to, however, is that you are not able to store VFS merged folders in the package without at least one virtual file inside. This fact is also likely the reason why Sequencer’s VFS –tab has behaved (and still behaves) maybe little bit unexpectedly when you remove files from it, suddenly removing folders from the mapping configuration when merged folder has been emptied.

I can only guess why this kind of design decision has been done when Softricity originally created VFS functionality in the SoftGrid (maybe Tim can shed a light on this matter?). By my understanding, VFS code in the App-V Client is one of the oldest and possibly much unchanged part since the original one, and maybe they wanted to save some bits in the SFT file; I really don’t know…

Not that it really matters that much, actually, since there’s not much point of having merged folders without some content inside it which is pretty much true for fully virtualized folders as well.

Another oddity of merged VFS folders comes from the fact that they are – by definition – intended to be merged with the local directory. In the past App-V versions, this meant that the local folder in the same exact path needed to exists, otherwise the merged folder or the contents thereof would not be seen at all by the virtual application (even the one coming from the package). I have witnessed this particular issue in the past myself.

In the more recent versions of App-V (I tested it with 4.6 RTM and 4.6 SP1, on Windows 7), the behavior of App-V has been changed – and I can pretty confidently say that this information has not been published by Microsoft in any release notes or any documents thus far – in a manner where App-V Client will silently create all non-existing merged folders over at the real file-system!

To see this in action and demonstrate exactly what I am talking about, let’s take our VFS test package and deploy it to the Win7 machine running 4.6 SP1 App-V Client. I have published Windows Explorer from the package so that we can browse to our C:\Program Files\Test Application and C:\Program Files\Test Application 2 directories, one of them being the fully virtualized and one merged.

But first, we will open Windows Explorer outside the package’s context to see how the file-system looks like before the package is opened:

That’s pretty standard set of directories in the empty Windows 7 machine and its Program Files -directory (only addition being the App-V’s Client). Note that the user account under which I’m conducting this test has no administrative rights to the system, so we are demonstrably not able to create new folders with it without credentials of a more privileged account:

Next, we start up our “bubble-enhanced” Windows Explorer from the package and observe what shall happen to the system. We can first test that our second Explorer instance is able to go to Q: drive so that we know it’s the one that is running inside our “VFS test” package’s environment:

After testing the virtual environment being around Explorer.exe and accessible to it, we browse over to the C:\Program Files with it and something immediately happens to the folder:

Right when the App-V Client tries to virtualize those two VFS folders over at %CSIDL_PROGRAM_FILES%, physical Test Application –folder appears in the C:\Program Files because otherwise App-V Client would not be able to merge virtualized Test Application with the local Test Application.

This all happens even when operating under limited user account and what’s more, the newly created directory (done from App-V Client’s security context, obviously, since there was no UAC consent dialog or explicit account details given by the user) is now having our limited user account set as the owner:

Pretty interesting, isn’t it?

While I agree that this feature handily gets around the issue that we did not have the folder pre-existing on the physical system (preventing VFS merging from happening), VFS folders virtualized outside user’s own profile triggering on-demand physical folder creation could potentially cause issues in multi-user environments as the new folder is now ACL’ed with initial user as the owner and normal user accounts having only read-access to it:

While it’s relatively rare to have merged folder outside the user’s own profile directory, which exists on the Sequencer but not on a Client machine, here’s yet another thing to look for when you are having VFS mappings in the package. If anything, it also stresses the best-practice that the Sequencing environment you have should not have things installed on it which your client environment does not, since in such a case merged VFS directories could appear in package outside the profile –related VFS directories.

And if such a directory ends up on a client machine, directory security may not go the way you intended when App-V Client creates those local folders for you, on-demand.

—

In the third and final installment of the series, we will turn our investigate eye into the fully virtualized (overridden) VFS folders and discover how they could leak information from the local system.

Stay tuned!