So VMware Horizon 6.2 was announced at VMworld just a week ago and the one feature I sorely wanted to see was automated provisioning (golden image management) of a Microsoft Remote Desktop Services farm.

The provisioning process is fairly straight forward, so in this blog post I’ll walk you through the steps to avoid any issues.

Prerequisites:

• Download the Agent, Connection Server and Composer software.
• Upgrade your Connection Servers to 6.2.
• Upgrade your Security Servers to 6.2 (remember you’ll need to repair with the connection servers).
• Upgrade your Composer.
• A Microsoft RDS server.

Once the upgrade is complete of the core infrastructure, lets hop to the Customisation Specification.

Creating a Customisation Specification:

So I originally missed this step completely and was left red faced after hours of troubleshooting and pestering some names in VMware (Sorry Ben, Pat, I’ll RTFM next time!).

I’m going to do this demo in the “God Awful” web client, but its the same in the GUI Console anyway.

Open the web console, and select Policies:

Select Customisation Specification > new:

Select Windows, Create a Logical name and description, then click next:

 Enter some organisational specific info, or not, who cares, click next:

On the Computer name Tab, for the sake of convenience, choose “use the virtual machine name”, click next:

Enter a product key if you wish, or rely on KMS, click next:

Enter the local administrators password for after the sysprep, then click Next:

• Select the relevant timezone, click next:
• Do not configure a “run once”, click next:
• Select which ever networking method you like, DHCP is probably easier,click next.
• Enter the domain name and credentials to domain join. Then Click Next:

Ensure to select “Generate New Security ID” then click Next:

Click finish on the Final page and ensure your new template shows (mine disappeared the first time, sneaky so and so):

Preparing the target:

Now lets hop first to the Remote Desktop Services target and install / upgrade the agent software!

Select the agent on the RDS server and kick off the installer:

Select Next, Agree to the license, choose IPV4, IPV6 if you’re crazy and click Next:

On the next screen, ensure to enable the “VMware Horizon View Composer Agent” and select Next:

after that, finish the install. Restart once, Shutdown and snapshot the VM.

Onwards!

Creating the Automated Farm:

Open the Horizon console and head to Resources > Farms > Add:

Select Automated Farm and click Next:

Select the vSphere server, select  Next:

Enter the Farm Name, click Next:

Enter the naming pattern for the farm servers, select Next:

Enter your Vsan details if you opted in to that sort of thing, click Next:

On the vCenter settings, choose the VM we configured in the target section earlier, select OK:

On the SnapShot selection, choose the Snapshot we created and click OK:

After that, just select whatever you want and click Next:

On the storage selection, choose your own settings then click Next:

Select The guest settings, including the Specification we created earlier:

Once finished, simply click finish and pray you’ve gotten everything right:

And that’s it! go have a coffee and check to ensure you have a VM after the deployment:

Now to create a pool:

Select catalog > Desktop Pools > Add:

Choose RDS Desktop Pool, Select Next:

• Name the Pool something useful, then select Next:
• Set the Pool settings, then select Next:

Select the newly created farm, then select Next:

Select “Entitle users after this wizard finishes” then select Finish:

Click Add, enter the group you wish and click OK once selected:

Click Close:

Now open the horizon client, tada!

To recap, below you will find the previous articles:

• ThinIO facts and figures, Part 1: VDI and Ram caching.
• ThinIO facts and figures, Part 2: The Bootstorm chestnut.

Off topic note:

two years ago at an E2EVC event, the concept behind ThinIO was born with just a mad scientist idea amongst peers.

If you are lucky enough to be attending E2EVC this weekend, David and I will be there presenting ThinIO and maybe, just maybe there will be an announcement. Our session is on Saturday at 15:30 so pop by, you won’t be disappointed.

Back on topic:

So here’s a really interesting blog post. Remote Desktop Services (XenApp / XenDesktop hosted shared) or whatever you like to call it. RDS really presents a fun caching platform for us, as it allows us to deal with a much higher IO volume and achieve deeper savings.

We’ve really tested the heck out of this platform for how we perform on Microsoft RDS, Horizon View RDS integration and Citrix XenSplitPersonality with Machine Creation Services.

The figures we are sharing today are based on the following configuration and load test:

• Citrix XenDesktop 7.6
• Windows Server 2012 r2
• Citrix User Profile Manager.
• 16gb of Ram.
• 4 vCpu.
• LoginVSI 4.1 medium workload 1 hour test.
• 10 users.
• VMFS 5 volume.

Fun figures!

Diving straight in, lets start by looking at the volume of savings across three cache types.

Reviewing the details for a moment:

Running repetitive tests of at least 3 per cache type, we found even at the lowest entry point we would support (50mb per user) we saw phenomenal savings of over 70% on write IO.

No pressure no diamonds!

To put that into perspective, at a 512 MB cache for 10 users, our cache reached maximum capacity at the second user login. With 8 users still left to login, cache full and still an hours worth of load testing left, our ThinIO technology was under serious pressure.

This is key to why ThinIO is such a great solution. We won’t just perform great until we fill our cache, we don’t require architecture changes or care about your storage type, we have no lead times or install days, we will carry on to work with what is available to use, to take a large ammount of pressure off storage IOPS and data throughput.

With the figures above, you can see just how well the intelligence behind our cache can scale even when it faces such a steep workload.

Below you will find a breakdown of each test:

512 MB cache:

Breaking down into the figures, on the 512mb cache test, it’s clear to see just how well ThinIO deals with the tiniest of caches:

When we side by side this with our baseline averages, you can see we take a huge chunk out of that Spiky login pattern and continue to  reduce the steady state IO as the test continues:

So lets move up and see how we get on!

1024 mb cache:

Doubling up our cache size we see a great increase in both read and write savings as you’d expect.

With 100mb of cache per user, and the average user profile in the test 3 times that size. We are still under pressure. As we will natively favour optimisations to write IO over read, you’ll see the bulk of improvements happen in write when we’re under pressure as illustrated in this test:

With more cache available during the peak IO point, we make further savings on write:

2048 mb cache:

and at our recommended value of 200mb per user in Remote Desktop Services, the results are phenomenal! With this size, even still below the 300mb mark per user profile, the read IO gets a really good boost and the write IO saving well over the 95% mark!

And the side by side comparison is every bit as good as the savings illustrated above, reducing that peak bursty IO to just 41 IOPS:

But there’s more! 

As i pointed out in the previous blog, IOPS are just one side of the story. A reduction of data throughput to the disk is also a big benefit when it comes to storage optimisation, and as you can see we make a big difference:

Wrap up:

So there you have it, with ThinIO, a simple, in VM solution, you can you seriously reduce your IO footprint, boost user performance and achieve greater storage density per virtual machine or on Remote Desktop Services technology.

In the mean time:

If you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

• ThinIO facts and figures, Part 1: VDI and Ram caching.

Getting right to it:

In this industry when somebody says ‘boot storms!’ – most of us will respond with:

Boot storms are a well documented, boring problem and have many solutions available from vendors and hypervisors alike. Most solutions today rely on a ‘shared memory’ storage area to cache ‘on boot’, in theory caching only one startup or one pattern in order to then serve it back to the proceeding desktops to boot.

But why are boot storms an issue? While working on ThinIO we had the unique ability to really dive into the Windows boot process and analyse why boot storms cause the damage they do and in this post we thought we’d share our findings to better document the issue.

Boot data:

Taking a typical windows 7 boot, to the login screen and idling until all services have started, the data traversing from disk to VM is relatively small. in our testing we found an average of Just 500-600 mb of data is read during this process, and write data barely registers at between 20 and 30mb.

But hey, what gives? Taking such low data throughput, why is boot such a contenscious issue? Have I been misled with marketing and vendor nonsense?

The IO chestnut:

Sadly no, it’s the way windows requests this data, but don’t take my word for it…. Behold, the incredible mess that is the Windows boot process!

Yep, that’s right, in the time Windows requested roughly 600mb of data, it sent down an astounding 70 thousand IO’s in the space of 2-3 minutes!

Math time:

Now if you were to take these figures as they stand, you would take 70,000 IO’s divide this into 560mb and you’d probably end up with an average of about 8k of data requested per IO… You’d be wrong.

As my good buddy Conor Scolard would say, ‘when you Assume, you make an ass out of you and me’.

To better understand the bounderies of Windows, Windows requests IO’s between the minimum of 512 bytes all the way up the spectrum later in the boot process to 128k and above. But it requests these blocks sparcely, on demand, and not just once per sector, the same blocks are frequently accessed.

The net result of this causes absolute havok on the storage:

The crux of the issue is, for each one of these IO’s, the storage provider needs to compute the block data requested, seek the data out, then return it.

But 70,000 of these IO operations for a meagre 600mb of data is madness and you can now see exactly why boot storms were labelled as such for those early adopters who had their hands burned by this fact finding mission.

I’ll mitigate this issue by just booting my VM’s at night!

I’m sure you will! I would also love to see your face if a number of users happen to restart their desktops during the day, cascading 70,000 IO’s per desktop to the storage in a 2 minute window, per desktop!

Bootstorming IS an issue.

Now, knowing all this, it makes sense as to why storage and hypervisors alike are using a cache of ram.

But how does ThinIO fit in here? With Read Ahead of course!

Knowing the Windows boot process as intimate as only a technology like ThinIO can, there are many, many optimisations we can make to this process.

We can both speed the boot process up and also massively reduce the storage requirement while in VM, without any fancy caching mechanism!

With ThinIO’s read ahead technology, we can deliver just shy of an 80% boot IO reduction with nothing other than having our technology in the virtual machine:

Taking a ThinIO averaged test and overlaying it to a baseline averaged test, it’s clear just how much impact this technology can have:

Wrap up:

So there you have it, with ThinIO, a simple, in VM solution, not only can you seriously reduce your IO footprint, boost user performance and achieve greater storage density per virtual machine, you also can also massively negate the impact a booting VM has on your storage.

If you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

Warm up your labs or fire up your golden images ladies and gents, we’re delighted to announce ThinIO’s brief public beta will begin today!

This project has taught us some really interesting things about Windows IO, how Windows behaves and how the hypervisor and storage can behave. This project really felt like a David vs. Goliath task as we (members of our community with a desire to simplify this issue) attempted to tackle one of the largest issues in our industry, storage bottlenecks and Windows desktops.

What’s really unique about our approach is there are no hardware lead times, no architecture changes needed and no external dependencies. ThinIO can be installed in seconds and the benefits are seen immediately.

We’ve spent countless hours testing, tuning, retesting and even more tuning. We’re extremely happy with the results. This public beta will serve as an opportunity for you to really kick the tyres and believe the hype in what we’ve built while we’re putting together the final touches to release the product in the coming weeks.

During this time, we found achieving positive and consistent IO negation boils down to a number of items:

• cutting down on the volume of IOPS sent to the storage.
• Reducing the data transferred (MB/sec) to and from the storage.
• Intelligently cutting down on peak IO, such as boot and user logon.

In the coming days we’re going drill down into these categories in more depth. But as a quick overview, here’s a baseline (top) and ThinIO (bottom) session comparison of a Windows 8.1 desktop login, 1 hour Login VSI medium workload and log off with just 350 mb of cache for ThinIO:

Keep an eye out for the coming blog posts, but in the mean time, the ThinIO beta is available to download here now! Go forth and have fun.

Until next time,

A

In this customers environment, they are using XenApp 6.5 on Server 2008 R2 for published desktops, this environment is a hosted desktop environment for a number of countries in Europe.

Infrequently an issue could be observed where the users timezones would shift out by one or two hours within the Lotus Notes application. This would case SameTime conversations and Calendar times to display out by the aforementioned value above.

When this issue occurred, it happened to all users on the server. A restart of the server did not fix the issue.

Interestingly, a “TZUtil /g” was reporting the client was in the correct time zone:

If you ran “TZUtil /s GMT Standard Time“, then closed and opened Lotus Notes… The problem was resolved for that user, in that session until they logged off.

It’s worth pointing out, that this issue was only seen in Lotus Notes, not in any other application, java or otherwise.

When comparing the TimeZone settings from a problematic server to a working server, I found the following difference:

These keys are stored under:

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlTimeZoneInformation

And the working server looked as follows:

Now that is weird! So we copied the correct keys from the server to server and the issue was resolved. On all servers once users closed and opened Lotus Notes again.

But what caused this?

With a work around in place, I began to dig deeper into what caused the timezone to change on the servers despite the fact that no users have the ability to do so.

Analysing the logins to the servers, I spotted an administrator account logging into each of the servers as the day went by. This user didn’t log into the correctly working servers so this was the first clue.

Now if you’ve used Lotus Notes combined with XenApp and timezones before, you’ll know its a complete nightmare, interestingly the administrator in question (me, shamefully), was logging onto a XenApp session with a linux timezone to replicate an issue.

More embarrassingly, I then decided to Remote Desktop inside of the XenApp session to the affected servers, and with my admin account being who it was… inadvertently changed the timezone for the servers it seems.

That doesn’t sound right? You rdp’d from a client in a different time zone and it changed the server timezone?

I agree, but I have since been able to replicate this in a test environment. As with Server 2008 Microsoft now handle the timezone redirection themselves as part of group policy and administrative accounts will change the timezone of the server intermittently.

Now most customers probably wouldn’t even notice this, unless they are using lotus notes, as all other applications behaved correctly.

How do you work around this issue?

Ensure that the Group Policy you use to configure timezone redirection is configured to “not apply” to any local administrator on the XenApp server that may log in.

Throughout this blog post, I’m going to be talking about Process Affinity and Process Priority. Understanding these definitions will help. I’m also only targetting server 2008 R2 and Windows 7 for this post.

An issue I see time and time again in Virtual Desktop Infrastructure and Server Based Computing environments is CPU spikes cause sessions to pause and stick. These CPU spikes can be caused by overcommitment or overzealous application usage and the vendors (Citrix, ThreadMaster, RES, Appsense, etc) have come to the rescue frequently with tools to reduce these events from occurring.

The problem with these solutions (Appsense excluded) is they are reactive, I.E. they take a number of seconds to identify a spike and respond by dropping the priority of the process or reducing the Users time on the processor.

These few seconds of high usage, reduce the amount of time the users applications and desktop session can spend on the processor, this reduction of resources cause the VDI desktop of the user to be completely unusable during the pause and ultimately resulting in complaining from the user community. Worse yet, if this is an SBC environment you now have all users on that server locked out!

And all this may sound a bit over the top, but its very easy to reproduce. For example, create a Microsoft Excel sheet and attempt to vlookup over a few thousand cells and watch the processor get chewed up. In this example, yes a vLookup across that many cells would be better served from a server based solution, but what’s stopping your users doing it?… Exactly.

Looking at SBC for a second:

With Server 2008 R2, the Fair Share CPU scheduling feature really is best (free) solution out there for CPU Utilisation management. The processor scheduling is far better than the Citrix equivalent and they also offer a feature (albeit, it feels unfinished) called Windows System Resource Manager to configure the processor affinity along with Process soft rules for performance management.

By setting these extremely intensive Multi-threaded applications to an affinity, we can “Lock” these processes to only run on certain processor cores, allowing the remaining cores to rebalance the rest of the workload. This ultimately, leaves the users session still responsive even if the application no longer responds. Allowing the user to continue working in other application until the intensive process is complete.

The problem with Windows System Resource Manager?

• It’s a pain to configure on more than one system
• It requires (yeuck) Windows Internal Database
• It has been marked as “Depreciated” in Server 2012
• If a user changes affinity, Resource manager doesn’t re adjust.
• It still suffers from “pauses” or sluggishness when the Cpu is really being hammered.

This was my dillema recently with both XenApp & Xendesktop. Ultimately not happy to Pay for a solution, embrace a depreciated tool, or wash my hands of the problem, I decided to write my own tool, complimenting fair share CPU scheduling but allowing affinity and priority locking.

Introducing Threadlocker:

ThreadLocker has been written to help you target known problematic processes and deal with them pro actively.

Threadlocker also has the added benefit of dropping the priority of known grunty processes to idle, meaning the process will use as much CPU as it can, but any other higher priority process can interupt without sluggishness or pauses.

With ThreadLocker you can target these processes and set their Affinity number of processors they can run on:

Or their Process Priority to drop them out of the normal running context:

• Threadlocker is light weight and scalable.
• Threadlocker works flawlessly with Cpu Fair Share Scheduling, so even if a process is “ThreadLocked” the users running those applications will fair share on the designated cores.
• Threadlockers configuration is xml based and can be copied down to the machine with Startup script or Group Policy preferences.
• Threadlocker can be used in VDI environments where no other free solution is available.
• Threadlocker will re-adjust priority, or affinity if the devious user tries to remove the restriction.

Threadlocker has been successfully tested on the following platforms:

• XenApp 6.5
• Windows Server 2008 R2 Service Pack 1
• Windows 7 x64 Service Pack 1

• Threadlocker Requires .Net Framework 3.5 sp1

Download:

Here

While working in a XenApp 6 proof of concept I came accross this little feature and decided its time to share it!

When a user right clicks on the desktop, by default they get access to commands to manipulate the appearance of the desktop. As I restricted access to the control panel, the two options below were generating errors in the users sessions:

The error generated is your standard group policy restrictions error message as below:

While digging into this further I found the following registry key that corresponds to the two prompts we see above.

HKEY_CLASSES_ROOTDesktopBackground

Under this key, you can see both entries that appear on the shell extension menu;

The problem with this key is, its owned by the TrustedInstaller account, and by default administrators cannot modify it. To modify this  key and hide this menu from users (but maintain it for administators) please follow the below steps.

Please note, any hotfixes from microsoft may remove your hard work, so be prepared to redo this work if Microsoft decide to work with this key in future.

1. Take a backup of this key, you’ll thank me if you get it wrong!
2. Browse down to desktopbackgroundshelldisplay
3. right click this key, choose permissions, click advanced then owner
4. Select administrators from the list, then choose “Apply”.
5. browse to the permissions tab and remove the “users” group.
6. Click “apply”, then “ok”.
7. The “screen resolution” menu should now disappear from any current and future sessions.
8. Repeat step 2 to 8 on DesktopBackgroundShellPersonalize.
9. Tada! go grab a coffee to celebrate your domination over the windows operating system.

And that’s it, you should now have a lean, clean and  error free shell extension menu when right clicking on the desktop.

Pedantic, begrudging scripters note:

Now if you’re a pedantic scripting so and so like me, you wont be satisfied to leave this job as a manual task. And despite spending more time than I’d like to admit, I couldn’t perform this work in powershell despite what I tried. Luckily the task was extremely easy to do with Helge Klein‘s setacl program.

Below is an example of a script to achieve this:

setacl.exe -on HKLMsoftwareclassesDesktopBackground -ot reg -actn setprot -op dacl:p_nc;sacl:p_nc -rec yes

SetACL.exe -on HKLMsoftwareclassesDesktopBackground -ot reg -actn ace -ace “n:system;p:read” -ace “n:administrators;p:read” -actn clear -clr “dacl,sacl” -actn rstchldrn -rst “dacl,sacl” -rec yes