To recap, below you will find the previous articles:

• ThinIO facts and figures, Part 1: VDI and Ram caching.
• ThinIO facts and figures, Part 2: The Bootstorm chestnut.

Off topic note:

two years ago at an E2EVC event, the concept behind ThinIO was born with just a mad scientist idea amongst peers.

If you are lucky enough to be attending E2EVC this weekend, David and I will be there presenting ThinIO and maybe, just maybe there will be an announcement. Our session is on Saturday at 15:30 so pop by, you won’t be disappointed.

Back on topic:

So here’s a really interesting blog post. Remote Desktop Services (XenApp / XenDesktop hosted shared) or whatever you like to call it. RDS really presents a fun caching platform for us, as it allows us to deal with a much higher IO volume and achieve deeper savings.

We’ve really tested the heck out of this platform for how we perform on Microsoft RDS, Horizon View RDS integration and Citrix XenSplitPersonality with Machine Creation Services.

The figures we are sharing today are based on the following configuration and load test:

• Citrix XenDesktop 7.6
• Windows Server 2012 r2
• Citrix User Profile Manager.
• 16gb of Ram.
• 4 vCpu.
• LoginVSI 4.1 medium workload 1 hour test.
• 10 users.
• VMFS 5 volume.

Fun figures!

Diving straight in, lets start by looking at the volume of savings across three cache types.

Reviewing the details for a moment:

Running repetitive tests of at least 3 per cache type, we found even at the lowest entry point we would support (50mb per user) we saw phenomenal savings of over 70% on write IO.

No pressure no diamonds!

To put that into perspective, at a 512 MB cache for 10 users, our cache reached maximum capacity at the second user login. With 8 users still left to login, cache full and still an hours worth of load testing left, our ThinIO technology was under serious pressure.

This is key to why ThinIO is such a great solution. We won’t just perform great until we fill our cache, we don’t require architecture changes or care about your storage type, we have no lead times or install days, we will carry on to work with what is available to use, to take a large ammount of pressure off storage IOPS and data throughput.

With the figures above, you can see just how well the intelligence behind our cache can scale even when it faces such a steep workload.

Below you will find a breakdown of each test:

512 MB cache:

Breaking down into the figures, on the 512mb cache test, it’s clear to see just how well ThinIO deals with the tiniest of caches:

When we side by side this with our baseline averages, you can see we take a huge chunk out of that Spiky login pattern and continue to  reduce the steady state IO as the test continues:

So lets move up and see how we get on!

1024 mb cache:

Doubling up our cache size we see a great increase in both read and write savings as you’d expect.

With 100mb of cache per user, and the average user profile in the test 3 times that size. We are still under pressure. As we will natively favour optimisations to write IO over read, you’ll see the bulk of improvements happen in write when we’re under pressure as illustrated in this test:

With more cache available during the peak IO point, we make further savings on write:

2048 mb cache:

and at our recommended value of 200mb per user in Remote Desktop Services, the results are phenomenal! With this size, even still below the 300mb mark per user profile, the read IO gets a really good boost and the write IO saving well over the 95% mark!

And the side by side comparison is every bit as good as the savings illustrated above, reducing that peak bursty IO to just 41 IOPS:

But there’s more! 

As i pointed out in the previous blog, IOPS are just one side of the story. A reduction of data throughput to the disk is also a big benefit when it comes to storage optimisation, and as you can see we make a big difference:

Wrap up:

So there you have it, with ThinIO, a simple, in VM solution, you can you seriously reduce your IO footprint, boost user performance and achieve greater storage density per virtual machine or on Remote Desktop Services technology.

In the mean time:

If you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

Here’s a really strange and interesting issue I faced late last week that resulted in a few head scratching moments and late nights.

An Issue began two weeks ago intermittently in a XenApp 4.5 farm used for hosted desktops, intermittently NonPagedPool bytes would shoot through the roof, the event logs would become inundated with event 333 errors and the servers would lock up completely.

The running sessions could no longer open new applications, performance was extremely poor and RDP’ing to the XenApp server would result in an RPC error. Disconnecting the sessions remotely would also result in an RPC error or TSAdmin was completely incapable of connecting to the server. We had no choice but to dump the servers using NMI and pray for a crash dump.

No changes had been made to the environment in a number of weeks and the last change to the environment was a “Citrix Ready” printer driver from Lexmark. As the days progressed the issue got worse and worse with more servers going offline day by day. Although we did initially catch a number of crash dumps, we hit a bad run of luck with most of them being corrupt on restart.

By day six, 9 servers went offline throughout the day, I was pulled in to assist resolve this massive issue.

I fired up the windows debugging tools and managed to get a look at a crash dump fresh from a locked up server.

Using !vm i pulled the virtual memory statistics at the point of the crash:





So we had a serious non paged pool leak as we suspected, but what exactly was chewing up all that nonpaged?

Running !poolused 2, i was able to drill down into the drivers using nonpagedpool and view which driver tag was using the largest amount of the pool as below:





reviewing the list, i was immediately alarmed by the amount of ram in use by the “Ica” pool tag. Having reviewed 100’s of memory dumps I had never seen the Ica pool tag listed in the top 20, never mind using 99721328 bytes (~95mb).

The Ica pool tag is fairly obvious as to who owns it, but just to be on the safe side and to drill down to the owning driver, I ran the following command on the drivers folder to find references to the “Ica” pool tag.



findstr /m /l Ica *.sys



So we got quite a few hits off the Ica pool tag. Quite a number of the above drivers are Microsoft’s, which is not suprising in the grand scheme of things as we all know the origination of the RDP protocol.

So with a little more information to hand, I set about googling this chain of events to see if it’s either documented, or hotfixed. A search yielded quite alot of articles including a private hotfix and a Rollup pack.

Drilling down into the technotes to see if I could find a potential cause for this issue, I was left a little wanting with the lack of information available:

Servers with Hotfix Rollup Pack 6 installed can run out of non-paged pool memory in the ICA Pool tag. The issue occurs when users log on to the server and its frequency increases with the amount of time since the most recent restart and the number of logons since the restart.

What irked me here, was the lack of information and the fact that these servers had been running HFRP 6 for roughly 18 months with no issues similar to this.



Why all of a sudden are we losing servers all over the place to such an issue?



I dug further into the hotfix notes with help from my good friend and all round cool Citrite James Denne, the hotfix specifically noted:

When a server is in low memory condition the <Redacted>() spins in an infinite loop by constantly allocating an OUTBUF and trying to write it on the stack. This problem happen when system is going in and out in low memory condition.

So there’s a great explanation of the issue from the horses mouth, but again there was a niggling problem in the back of my head…



These servers weren’t spinning in and out of low memory, our pool usage reporting would have caught this?



I was satisfied to see a hotfix was available, but in the back of my head I was concerned about the change that may have caused this issue, it’s still unclear what is causing this low memory condition to spin the infinite loop and why we couldn’t see the low memory scenario before it happens. Being an massive issue, we had to make a quick turn around here. We had a choice of going to HFRP 7 or using the private hotfix available. I chose the private hotfix, for two reasons:

• Mass Deploying a roll up pack to fix one problem is like tapping a nail in with a sledge hammer.
• My experience with HotFix Rollup Packs is they fix your issues, but introduce at least one new one.

We took all the servers offline for emergency maintenance that night and cautiously waited for the morning to come and see if our issue was resolved.



and so we patiently waited…



Once hotfixed and rebooted, we arrived at the office early to watch as the user sessions began to filter in to the Farm. All was quiet for the first hour or so, but then the phones started.

once the user load hit 15-16 users per XenApp session, a number of servers began to again log a number of eventlog 333’s as below:






Dammit.



Frantically we connected to the console of a server, to check the paged pool states but again no alerts on pagepool size? as below the ICA pool tag was nowhere to be seen:





And the ica tag was at a much more respectable / expected value as below:





Our next clue came in the form of the following, when users were logging in they were getting the following error:






So we’ve fixed our Ica memory leak, now what the hell is happening?



If memory usage for the pools are ok but we’re still getting errors about flushing to the registry, and now new user profiles can’t load their profiles, my hunch was there had to be something wrong with the registry hives…

I used command prompt to open the “Documents and Settings” folder and ran the following command:

dir /s /a ntuser.dat

With a quick glance, i found the following:





The “Citrix Print Manager Service” user account had a registry hive of over 50mb? What in the name of superman is hiding in that registry hive?

To rectify the issue immediately, we stopped the above print manager service and forced the hive to be unloaded with delprof. Once we had done this, the user profiles began to load again on each affected server. But we’re now unable to use client redirected printing.



To regedit!



I mounted the registry of a profile that had failed to delete and drilled down to see what all the fuss was about. As this was  now firmly in the printing land, I went looking for keys to match the Lexmark driver change from a number of weeks ago.

What I found was extremely interesting, for each client redirected printer ever mapped with the Lexmark driver, we had an entry under both the PCLPlugin and PSPlugIn keys:





Although this was a really questionable practice from lexmark, I examined the keys under each entry for the PclPlugin key and they consisted of just a few small binary files of which were no more than a KB or two.

Upon looking at the same keys under PSPlugin, I found a new key, called GDL. This GDL key was absolutely massive and there was one for each and every time a client printer had been redirected using the Lexmark V2 driver.





I exported both both the users hive, and the psplugin key to text and the comparison is below:





The GDL key itself was over 3mb per redirected printer!?!?:



So there we have it.



The route cause was as follows:

This Lexmark driver has a weird tendency to store an unbelievable amount of crap in the registry for users.

The Citrix print manager service also suffers this faith when it maps a redirected printer.

As more and more users were testing in production (GRRRR!) / beginning to use a new printing solution on a customer site, this registry file began to grow and grow ultimately flooding the maximum registry size of 30% of the paged pool ram.

As the registry hive size was growing out of control, the Ica driver ran into a low memory situation and ultimately caused the infinite loop.

The Ica loop and nonpaged saturation was masking the printer driver bloat in the registry.

As the days went on, more and more servers began to saturate the Maximum registry size and go offline.

Corrective actions:

• Enforce a policy to not allow native drivers, in any way, shape or form when redirecting printers where possible.
• Obtain the latest driver from Lexmark is you have lexmark printers.
• Give lexmark an earful for not testing their drivers.

Lessons Learned:

• Don’t test things in production.
• Don’t trust a vendor with “Citrix Ready”, it’s up to them to test these things and they regularly don’t.
• Create a monitor for registry size (perfmon > system > % Registry quota in use)
• install the debugging tools on the XenApp 4.5 servers as this issue is going to become more prevalent. *

* This isn’t going to get any better.

As Vendors move further and further towards 64 bit architectures they can and will forget about the extremely restrictive memory sizes available in 32 bit versions of windows, 64bit windows has so much memory available for the pools they can be as sloppy as they want without much concern.

Server 2003, Windows XP and XenApp 4.5’s death bells are knelling and have been for some time.

You are going to see pagepool’s floods and other such nasties more and more in the coming months before you finally decommission your old server 2003 environment. My advice to you is to:

• get very comfortable with the following tools:
  • PoolMon.
  • Process explorer.
  • Windows debugging tools.
• Have a good read of the following article: 333, read it, read it again.
• Never be afraid to have a look at a dump file yourself.
• Throw an issue at every vendor possible during troubleshooting, it’s in their interest to prove it’s not their software at fault.
• Understand your pagepool sizes and limitations.
• Never trust a printer driver.
• Never, ever, ever trust a Vendor to behave accordingly or follow the Citrix Ready standards.
• If you absolutely, positively need to run something in server 2003 or XP, consider using XenDesktop hosted apps to isolate the problem to a singular kernel away from the bulk of your task workers.

Having been in this situation recently, I needed to audit and report on the types of files open on the file server, my hunch was a certain select number of users were running applications (like *gulp* lotus notes) from the network share.

Disappointed with the powershell scripts on the interwebs, I decided to write my own function to perform this task:

[sourcecode language=”powershell”]
function get-openfiles{
param(
$computername=@($env:computername),
$verbose=$false)
$collection = @()
foreach ($computer in $computername){
$netfile = [ADSI]"WinNT://$computer/LanmanServer"

$netfile.Invoke("Resources") | foreach {
try{
$collection += New-Object PsObject -Property @{
Id = $_.GetType().InvokeMember("Name", ‘GetProperty’, $null, $_, $null)
itemPath = $_.GetType().InvokeMember("Path", ‘GetProperty’, $null, $_, $null)
UserName = $_.GetType().InvokeMember("User", ‘GetProperty’, $null, $_, $null)
LockCount = $_.GetType().InvokeMember("LockCount", ‘GetProperty’, $null, $_, $null)
Server = $computer
}
}
catch{
if ($verbose){write-warning $error[0]}
}
}
}
Return $collection
}
[/sourcecode]

The function above (get-openfiles) has been written to accept an array of servers to the command line and it will return the following items:

• The ID of the open file.

• The server it’s open from.

• The username who has the file open.

• The amount of locks the file has.

A couple of quick examples for using this command are below:


Retrieving open files from server1:






[sourcecode language=”powershell”]get-openfiles -computername server1 | select server,itempath,lockcount[/sourcecode]



Retrieve a count of open files that end with the nsf file type (Lotus Notes):






[sourcecode language=”powershell”](get-open files -computername server1,server2 | ? {$_.itempath -like "*.nsf*"}).count()[/sourcecode]



Retrieve a report of total open files on a number of file servers:



[sourcecode language=”powershell”]get-openfiles -computername server1,server2,server3,server4,server5 | group -property server[/sourcecode]

“Your XenApp servers have very high disk queue’s and IO”

“What’s causing it?”

“dunno…”

With Server 2008, the task manager’s resource monitor feature will help you find these items. But in server 2003 this was a perilous task. The specific details for disk io per process are stored in performance monitor under each specific process running. Trying to analyse each process was a massive pain, but powershell can do some very clever work to help alleviate this!

I wrote two quick functions which act similar to “top” in linux for giving an on screen view, updating at interval of what exactly is creating IO activity. These two functions are:

get-IODataBytes:

Get-IODataOperations

The code for these functions are below:

[sourcecode language=”powershell”]
function get-iodatabytes{
$result=(get-counter -counter "Process(*)IO Data Bytes/sec" -ea 0).countersamples | ? {$_.cookedvalue -gt 0} | select instancename,@{Name="SessionID";Expression={if ($_.path.contains("#")){($_.path.split("#)"))[1]}else{"0"}}},@{Name="IO Data Bytes/sec";Expression={[math]::Round($_.cookedvalue,0)}},@{Name="IO Data KBytes/sec";Expression={[math]::Round($_.cookedvalue / 1024,0)}} | sort -Descending "IO Data Bytes/sec" | ft
$currentqueue=(((get-counter -counter "PhysicalDisk(0 C:)Current Disk Queue Length" -ea 0).countersamples) | select cookedvalue).cookedvalue
clear
write-warning "Hit [CTRL] + [C] to exit live capture"
write-host "Current Disk queue: $currentqueue"
return $Result
}

FUnction get-IODataOperations {
$result=(get-counter -counter "Process(*)IO Data Operations/sec" -ea 0).countersamples | ? {$_.cookedvalue -gt 0} | select instancename,@{Name="SessionID";Expression={if ($_.path.contains("#")){($_.path.split("#)"))[1]}else{"0"}}},@{Name="IO Data Operations/sec";Expression={[math]::Round($_.cookedvalue,0)}} | sort -Descending "IO Data Operations/sec" | ft
$currentqueue=(((get-counter -counter "PhysicalDisk(0 C:)Current Disk Queue Length" -ea 0).countersamples) | select cookedvalue).cookedvalue
clear
write-warning "Hit [CTRL] + [C] to exit live capture"
write-host "Current Disk queue: $currentqueue"
return $Result
}

[/sourcecode]

if you wish to loop one of these functions, simply use the following code:

[sourcecode language=”powershell”]
while ($true){
get-iodataoperations
start-sleep 1
}
[/sourcecode]

Hello there, and thank you for dropping back for part 3…

I suppose I should start with the disappointing news that I have yet to test this option for VDI in a box. And despite Aaron Parker’s suggestions it wasn’t due to lack of inspiration, it was down to lack of time! This series has gathered allot of interest from both community and storage vendors alike, and I feel I should set the record straight before I got any further:

1. This isn’t a production idea, you would be crazy to use this idea in a live environment.
2. Throughout this entire project, we’re focusing on pooled stateless. Stateful desktops would be a separate post entirely.
3. This wasn’t an attack on products in this market space, merely a fresh view on an old problem.
4. If i had the skills or funds necessary to run this project to a production solution, I wouldn’t have posted it. I would already be hard at work creating a reasonably priced product!

Now that my declarations are out of the way, I’d first like to talk about the moral of the story. This isn’t an unfamiliar expression:

IOPS mitigation is not about read IOPS it’s about WRITE IOPS!

VMware, Citrix and Microsoft have similar but very different solutions for read IOPS negotiation. Similar in the sense that they try to negate storage read IOPS. But the key difference with XenServer is the local disk cache via Intellicache has the out of box functionality to cache the majority of read’s to local disk (think SSD*) without the baked in soft limit of 512 MB for Microsoft HyperV and VMware respectively.

Long story short, VMware and Microsoft’s solution is about 512mb of recognizable read IOPS negation un-tuned, but enabled. Of course this value can be tuned upwards, but the low entry point of cache would suggest, at least to me, that tuning up will have an upsetting affect on the host.

This to me is why IntelliCache has the upperhand in the (value add product) VDI space for read IOPS negation and they even throw in the Hypervisor as part of your XenDesktop licensing, so win win, but what about those pesky write IOPS?

Let’s look at Citrix for a moment!

If we review article 1 and article 2 for a moment, the half baked idea formed by Barry Schiffer, Ingmar Verheij, Kees Baggerman, Remko Weijnen and I, when combined with Citrix IntelliCache turned out to be a phenomenal combination to create an IOPS avoidance product. But the key point we had in the back of our heads was, Citrix provisioning server already has this driver!

Citrix provisioning server has a RAM cache driver, with configurable size baked into the Provisioning Services product. This driver works in a very similar way to the EWF driver, just without the API and flexibility of a Microsoft driver.

There are customers of Citrix out there using RAM caching with XenApp who I spoke to, they have assigned a large (8gb+ cache, 3-4gb of cache utilised day to day) to negotiate the chance of a spillover, but it could still happen and this is an acceptable risk to their implementation. I struggled to find a XenDesktop customer using this method who would talk about their implementation.

But with XenDesktop in mind, using an overly large cache size to avoid spillover just doesn’t scale the way a consolidated Server Based Computing model would… Which leaves us back in this same dilemma, I’d like to cache in RAM, but I don’t want to risk users losing data when they perform a task we have not factored for.

So with all this in mind, lets talk about my final hurdle I faced with this project.

RAM cache flood and my half baked solution:

The Microsoft EWF driver suffered the same problem as the Citrix provisioning server when the cache filled, i.e. it froze or outright bug checked if you tried to reference a file put into the cache before you filled it with other information.

Jeff Silverman has done a great article on what happens in this event with the Citrix Provisioning Services RAM cache and I urge you to go read it so I don’t have to repeat it! Go now, I’ll wait right here for you to get back!

Read that article? Ok good, let’s continue.

To combat this scenario with the Windows EWF driver, I created a very rough around the edges windows service using polling (Sorry Remko) to check the size of the cache periodically via the EWFAPI and write the cache out to disk.

The function in the EWFAPI called EwfCommitAndDisableLive allowed me to dump the ram cache to disk and subvert the cache, going straight to disk when this event had occurred. By using this routine the ram cache simply disables itself and allows pass-through to disk at this point.

With this in mind, I tuned my service to spill to disk when the cache was greater than 80% of ram available, not in use by the operating system when the system booted, This worked well to a point, but the key failure to this approach became apparent when you opened a large number of applications and they struggled to occupy the space available around the cache.

My second attempt however, proved much more fruitful where I monitored for free system bytes in memory and if this dropped below a certain threshold the EWF drive began it’s dump routine to disk. Once the dump routine was complete, ram cleared and the writes had been committed to storage where the disk continued to use it for the remainder of the session. Once this spill over had occurred, a notification bubble fired in the user session warning them of the degradation of service and they should log off saving work at the next convenient moment…

Voila! no blue screen, spill to disk and user notification of degradation.

This wasn’t fool proof, it was very rough, it didn’t handle files being written larger than the RAM cache, but In my opinion, it satisfied the biggest fear and business case against using the Citrix Provisioning server ram caching, the ram cache flood scenario. I was happy with the results and it scaled well to the size of my lab, it showed what is possible with a RAM filter driver and it allowed me to prove my point before I started to poke the hornets nest of the storage market. So let’s park the EWF story for now and focus on my favorite subject, Citrix technologies.

Note: I won’t be making this service and driver solution publicly available, it’s not a production ready solution and Microsoft would sue the pants off me. I’m sure you’ll understand why, but if you want more information drop me an email.

The next part’s of this blog are all theoretical, but I know certain people I want to listen, are listening (Hi Kenneth, hope you are well :)).

Negating the negated. Lets talk about that spill over.

But what about that Spill over event? by using a spill over from ram to disk, we create a situation where we could change a steady, slightly reliable IOPS per from each desktop, to a situation where, “couldn’t a collection of spillovers at the same time cause your storage to becoming swamped with IOPS?”

Absolutely, and I’ve been thinking about this quite a bit… But there’s another hidden potential here even in a RAM spill over scenario…

With a little bit of trickery couldn’t we also negate this spillover event with a simple provisioning job?

With a bit of time spent thinking about this scenario, this is what I came up with…

Why doesn’t the controller, for XenApp or XenDesktop (PVS / DDC) copy or (my personal preference) create a local, blank differencing disk when a VM boots?

The hypervisor could be completely agnostic at this point, we could spill over to local disk, keeping write IOPS completely away from the shared storage? even in a spill over event?

This approach (in this half baked enthusiasts view) would negate the negated… But don’t take my word for it, lets go for some theoretical solutions.

Solution A: Implementing a spill over routine with the Citrix provisioning server driver.

So with my success with the Microsoft driver, I got myself thinking how easy would this be to do utilizing the Citrix Provisioning Services driver? Without access to the code, I’m going to be making slightly risky statements here, but I have allot of faith in Citrix that they could make this happen.

From everyone I have spoken to about this idea, they all see the value in the ability to spill out of RAM… So Citrix, please make it so. Below are some idea’s for deployment methods assuming Citrix do march down this route and the pro’s and con’s I see that live with each scenario.

Bear in mind, I’m not a storage guy, a full time developer or a software architect, I’m just an enthusiast that see’s potential, so drop your comments below as to what you think!

Machine Creation Services.

Idea A: Provisioning services improved RAM caching, Machine Creation services and Intellicache on XenServer.

Utilizing XenServer and MCS we could leverage the intellicache to negate the read IOPS as we proved in my own EWF driver testing, but still deliver on that spill over mechanism allowing continuation of service.

Pros:

• Read IOPS negation.
• RAM caching to remove write IOPS (bar a spill over).
• Continuation of service in a cache flood scenario.

Cons:

• Limited to XenServer.
• Over provisioning of RAM necessary per desktop.
• RAM spillover will result in a large amount of IOPS to the shared storage.

Idea B: Provisioning services improved RAM caching, Machine Creation services and Intellicache on XenServer… With local copy!

Same benefits as previous, but now, we have zero reliance on the shared storage when the VM is up (except for ID disk actions).

Pros:

• Read IOPS negation.
• RAM caching to remove write IOPS.
• Uses local resources in a spill over.
• Continuation of service in a cache flood scenario.

Cons:

• Limited to XenServer.
• Over provisioning of RAM necessary per desktop.

So that’s MCS in a nutshell with per VM caching, I think this solution has so much potential I can’t believe it’s not been done, but I digress. So lets park that topic for now and move on to Citrix Provisioning Services.

Citrix Provisioning Services:

So lets look at the “favorite of many” technology.

Idea A: Citrix Provisioning Services and Improved RAM caching driver.

In a pure Provisioning services environment, we would force our read IOPS via the lan, instead of storage protocol but still deliver a spill back to disk to allow continuation of service.

Pros:

• Hypervisor agnostic.
• RAM caching to remove write IOPS (bar a spill over).
• Continuation of service in a cache flood scenario.
• Potentially no shared storage needed, at all, if caching on the PVS server.

Cons:

• Read IOPS aren’t really negated, they’re just forced over another technology.
• Over provisioning of RAM necessary per desktop.
• RAM spillover will result in a large amount of IOPS to the shared storage / pvs server.

Idea B: Citrix Provisioning Services and Improved RAM caching driver.. With local copy!

Taking the above benefits, but with the gain of utilizing local storage in the spillover event.

Pros:

• Hypervisor agnostic.
• RAM caching to remove write IOPS.
• Uses local resources in a spill over.
• Continuation of service in a cache flood scenario.
• Potentially no shared storage needed, at all.

Cons:

• Read IOPS aren’t really negated, they’re just forced over another technology.
• Over provisioning of RAM necessary per desktop.

So lets Review:

And there we have it, 4 solutions to IOPS negation utilizing the Provisioning server RAM caching driver, with a little bit of a modification to deliver a robust solution to RAM caching.

The copy and creation of differencing disks again would deliver additional benefits to leverage the hardware you put into each Hypervisor without the Shared Storage investment.

Win Win, but is it?

There’s an oversight here:

There’s a niggle here that’s been bothering me for some time and you’ll probably note I mentioned it as a CON to most of the solutions above… I’m going to lay it out on the table in complete honesty…

“Isn’t over provisioning RAM on a per desktop basis a waste of resource? Wouldn’t it be better if we could share that resource across all VM’s on a Hypervisor basis?”

You betcha! If we are assigning out (for argument sake) 1gb of ram cache per VM, that RAM is locked into that VM and if another machine spills, the free RAM in other desktops is wasted.

You would be completely insane not to reserve this RAM per VM, if an over-commit for the VM’s is reached this RAM will merely spill out to a page file type arrangement, negating all your benefits.

Ultimately, assigning RAM in this way could be seen as wasteful in the grand scheme of things…

But there are other ways to skin this cat!

So this leads me on to something I was also considering which popped up in a twitter conversation recently. RAM disk’s and Hypervisors.

Current storage vendors will create a storage pool, consisting of ram inside a VM, per hosting hypervisor for local storage to allow good IOPS leveraging that VM. This VM can perform additional RAM optimizations like compression, de-duplication and sharing of similar pages to reduce the count.

This approach is super clever, but, In my humble opinion (please don’t kill me vendors), this approach is wasteful. Running a VM as a storage repository per host has an overhead to run this virtual machine and takes away from the agnostic solution that is possible…

What If the hypervisor provides the RAM disk?

So excluding ESXi for a second, as getting a RAM disk into that platform would require VMware to allow access to the stack. Lets look at Hyper-V (2012) and XenServer for a second…

With Unix and Windows platforms, RAM disks have been available for years. They were once a necessity and a number of vendors still provide them for high performance IO environments.

Lets say (for arguments sake) Citrix and Microsoft decide to provide a snap-in to their hypervisor to allow native RAM disks (or a vendor writes one themselves!) and maybe, they even decide to provide RAM compression, Spill over to local disk, dedupe, and page sharing on this volume from the hypervisor stack…

Wouldn’t this extend provide all the benefits we’ve spoken about, without the need for a VM per host? And using Thin Provisioning allow all desktops to share the large pool of RAMdisk available?

Yes, yes, it would.

Above is an example of how I would see this working.

So random pictures are fine and all, but what about the read IOPS negation technologies? and what about combining these with XenServer or Hyper-V?

XenServer and IntelliCache:

Well there you have it now, all the benefits of a per VM filter, leveraging intellicache for reads and spilling out to local disk… RESULT!

Pro’s:

• Read IOPS negation
• Write IOPS negation
• No Shared Storage required for running VM’s
• Shared pool of RAM for all desktop’s to use.

Con’s:

• A small one, but no migration of VM’s.

Provisioning server?

and again, all the benefits of a per VM filter, reads redirected via lan and spilling out to local disk… RESULT!

Pro’s:

• Write IOPS negation
• No Shared Storage required for running VM’s
• Shared pool of RAM for all desktop’s to use.

Con’s:

• A small one, but no migration of VM’s.
• No real read IOPS negation.

And HyperV + CSV Cache?

Well here’s an accumulation of my thoughts:

So lets just talk for a second about what we are seeing… Utilizing a RAM disk with spill over, and copied vhd’s on boot we are removing the need for shared storage completely and hosting the IOPS from the hypervisor, natively without the need for additional VM’s.

And see that little icon down the bottom? Yep, that’s right, live migration from host to host thanks to Microsofts Shared Nothing Live Migration!

Pro’s:

• Some write IOPS negation
• No Shared Storage required for running VM’s
• Liv migration.
• Shared pool of RAM for all desktop’s to use.
• write back to local disk.

Con’s:

• No full read IOPS negation.

Review.

I’m sure there’s loads of reading in here, and there will be tons of thought and questions after this blog post. This has been in my head for roughly 9 months now and it feels victorious to finally get it all down on paper.

At the end of the day, Ram Caching is the way of the future, you pay no maintenance on it, you keep your IOPS off your storage and with a little magic from Microsoft, or particularly Citrix. You could see these benefits.

If / When these technologies make it out to you, you could quite happily stick with Machine Creation services, leverage intellicache and your shared storage requirement could be a cheap NAS from any vendor. Provisioning services also see’s allot of benefits from this approach, but the real creme de la creme is in that intellicache feature.

The key functionality to these approaches is simple:

• Cache RAM
• Spill to disk
• Keep the differences on local storage if it spills.

One last thing?

In true Mark Templeton fashion, you know I have a one last thing. This actually hit me today while writing this post and to be honest, I’m so amazed by the potential of this idea I’m going to build it in the lab this weekend.

But until then, a teaser…

What if I told you there was already an overlay, freely available baked into a Microsoft technology that would allow you to overlay local storage in Hyper-V with RAM. A powerful API attached to it, and the ability to write specific files to the local disk, subverting the overlay only when needed?

• RAM disk type technology? > yep
• already available? > yep
• Powerful API? > yep
• already can spill to disk? > yep

Yep Yep Yep! check back in a few days.

(If you know what it is already, keep it to yourself until I’ve gotten out of the lab!)

What do you think?

It doesn’t take a half baked enthusiast like me to see this potential and I’d be really eager to hear your comments on these approaches. If you would prefer to keep your comments offline, you can reach me on andrew@andrewmorgan.ie.

In this customers environment, they are using XenApp 6.5 on Server 2008 R2 for published desktops, this environment is a hosted desktop environment for a number of countries in Europe.

Infrequently an issue could be observed where the users timezones would shift out by one or two hours within the Lotus Notes application. This would case SameTime conversations and Calendar times to display out by the aforementioned value above.

When this issue occurred, it happened to all users on the server. A restart of the server did not fix the issue.

Interestingly, a “TZUtil /g” was reporting the client was in the correct time zone:

If you ran “TZUtil /s GMT Standard Time“, then closed and opened Lotus Notes… The problem was resolved for that user, in that session until they logged off.

It’s worth pointing out, that this issue was only seen in Lotus Notes, not in any other application, java or otherwise.

When comparing the TimeZone settings from a problematic server to a working server, I found the following difference:

These keys are stored under:

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlTimeZoneInformation

And the working server looked as follows:

Now that is weird! So we copied the correct keys from the server to server and the issue was resolved. On all servers once users closed and opened Lotus Notes again.

But what caused this?

With a work around in place, I began to dig deeper into what caused the timezone to change on the servers despite the fact that no users have the ability to do so.

Analysing the logins to the servers, I spotted an administrator account logging into each of the servers as the day went by. This user didn’t log into the correctly working servers so this was the first clue.

Now if you’ve used Lotus Notes combined with XenApp and timezones before, you’ll know its a complete nightmare, interestingly the administrator in question (me, shamefully), was logging onto a XenApp session with a linux timezone to replicate an issue.

More embarrassingly, I then decided to Remote Desktop inside of the XenApp session to the affected servers, and with my admin account being who it was… inadvertently changed the timezone for the servers it seems.

That doesn’t sound right? You rdp’d from a client in a different time zone and it changed the server timezone?

I agree, but I have since been able to replicate this in a test environment. As with Server 2008 Microsoft now handle the timezone redirection themselves as part of group policy and administrative accounts will change the timezone of the server intermittently.

Now most customers probably wouldn’t even notice this, unless they are using lotus notes, as all other applications behaved correctly.

How do you work around this issue?

Ensure that the Group Policy you use to configure timezone redirection is configured to “not apply” to any local administrator on the XenApp server that may log in.

Previously when installing this in server 2003, I just used the tried and trusted script from the Scripting guys here and it went off without a hitch. But with windows vista and above interactive session detection foils our lovely plans.

So moving on, what other options do we have? You could use:

• The managed API AddFontResource (nope, type 1 fonts dont register the same way as typical fonts)
• Using Visual Studio to create an MSI: (nope, see above)
• Using a third party msi builder: (nope, see above)
• Copy the files from the fonts directory from one machine to another (nope, something is missing)

So, after much hair pulling I hit a dead end.

I’ll spare you the details, but here’s an easy way to do it going forward:

• install the font on a test machine:
• Open command prompt as an administrator and browse to c:windowsfonts
• find the font files here and copy them out to a working directory

• Open regedit and browse down to:

hklm:softwaremicrosoftWindows NTCurrentVersiontype 1 installertype 1 Fonts:

• Export this key to your working directory:

Move to the target machine:

• Copy the fonts from the working directory to the fonts folder as above.
• Import the registry key.
• Restart the machine.

Tada!

If you know of an easier way to do this, be my guest to share it. But I dont want to look at a font file again for a very long time.

Throughout this blog post, I’m going to be talking about Process Affinity and Process Priority. Understanding these definitions will help. I’m also only targetting server 2008 R2 and Windows 7 for this post.

An issue I see time and time again in Virtual Desktop Infrastructure and Server Based Computing environments is CPU spikes cause sessions to pause and stick. These CPU spikes can be caused by overcommitment or overzealous application usage and the vendors (Citrix, ThreadMaster, RES, Appsense, etc) have come to the rescue frequently with tools to reduce these events from occurring.

The problem with these solutions (Appsense excluded) is they are reactive, I.E. they take a number of seconds to identify a spike and respond by dropping the priority of the process or reducing the Users time on the processor.

These few seconds of high usage, reduce the amount of time the users applications and desktop session can spend on the processor, this reduction of resources cause the VDI desktop of the user to be completely unusable during the pause and ultimately resulting in complaining from the user community. Worse yet, if this is an SBC environment you now have all users on that server locked out!

And all this may sound a bit over the top, but its very easy to reproduce. For example, create a Microsoft Excel sheet and attempt to vlookup over a few thousand cells and watch the processor get chewed up. In this example, yes a vLookup across that many cells would be better served from a server based solution, but what’s stopping your users doing it?… Exactly.

Looking at SBC for a second:

With Server 2008 R2, the Fair Share CPU scheduling feature really is best (free) solution out there for CPU Utilisation management. The processor scheduling is far better than the Citrix equivalent and they also offer a feature (albeit, it feels unfinished) called Windows System Resource Manager to configure the processor affinity along with Process soft rules for performance management.

By setting these extremely intensive Multi-threaded applications to an affinity, we can “Lock” these processes to only run on certain processor cores, allowing the remaining cores to rebalance the rest of the workload. This ultimately, leaves the users session still responsive even if the application no longer responds. Allowing the user to continue working in other application until the intensive process is complete.

The problem with Windows System Resource Manager?

• It’s a pain to configure on more than one system
• It requires (yeuck) Windows Internal Database
• It has been marked as “Depreciated” in Server 2012
• If a user changes affinity, Resource manager doesn’t re adjust.
• It still suffers from “pauses” or sluggishness when the Cpu is really being hammered.

This was my dillema recently with both XenApp & Xendesktop. Ultimately not happy to Pay for a solution, embrace a depreciated tool, or wash my hands of the problem, I decided to write my own tool, complimenting fair share CPU scheduling but allowing affinity and priority locking.

Introducing Threadlocker:

ThreadLocker has been written to help you target known problematic processes and deal with them pro actively.

Threadlocker also has the added benefit of dropping the priority of known grunty processes to idle, meaning the process will use as much CPU as it can, but any other higher priority process can interupt without sluggishness or pauses.

With ThreadLocker you can target these processes and set their Affinity number of processors they can run on:

Or their Process Priority to drop them out of the normal running context:

• Threadlocker is light weight and scalable.
• Threadlocker works flawlessly with Cpu Fair Share Scheduling, so even if a process is “ThreadLocked” the users running those applications will fair share on the designated cores.
• Threadlockers configuration is xml based and can be copied down to the machine with Startup script or Group Policy preferences.
• Threadlocker can be used in VDI environments where no other free solution is available.
• Threadlocker will re-adjust priority, or affinity if the devious user tries to remove the restriction.

Threadlocker has been successfully tested on the following platforms:

• XenApp 6.5
• Windows Server 2008 R2 Service Pack 1
• Windows 7 x64 Service Pack 1

• Threadlocker Requires .Net Framework 3.5 sp1

Download:

Here

I have to admit, I’m a bit torn with Windows 8 in general. I’m absolutely in love with Windows Server 8’s new Powershell functions and management console, but despise the lack of a start menu. Luckily Powershell has gotten so powerful in Server 8, I hope to not spend much time in the Gui.

Back on topic: By default when you install Windows Server 8 Core and log into the console, you get presented with a CMD prompt… Weird eh?

Now most administrators will simply type powershell and perform their tasks, but I personally feel this is the wrong way around. Powershell should launch first, and if cmd is really needed you could call it inside of powershell!

Being the pedantic individual that I am, I set about changing Server Core to auto-load powershell on login. This was quite an easy task and I’ve documented it below for other users to follow if they wish:

The shell in Windows Server 8 is configured under the following key:

HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionWinlogonAlternateShellsAvailableShells

Under this key, on Server 8 Core, you will see the following entry:

And that’s where our CMD is coming from! This is another one of these blasted TrustedInstaller Keys. So if you want to change it, you’ll need to take owership of the key, then assign full control to your user account.

So the first thing I did was create a value lower than 30000 and assign it to powershell, but this didnt work.

Unusually here the weighting system is highest wins, rather than lowest which is a little counter intuitive, but I digress.

I added a new key 90000 (i’ll explain this later) and entered the path to powershell as below:

Now Once I restarted, Powershell is the automatic shell of choice:

Bonus 1: Why 90000?

Well if you look at a Server 8 Gui server, you’ll notice that Explorer.exe wins the selection by being 60000:

So I added another 30000 and bob’s your uncle.

If you asked yourself, “hmmm, could i also do this on windows server 8 where the gui is installed?” the answer is yes.

And if you asked yourself  “hmmm, could I also do this on Windows 8?” the answer is no.

And if you also asked yourself “hmmm, could i assign these in a users key instead of local machine?” the answer is no.

(Note: this will also work with Windows 8 Consumer Preview)

While troubleshooting an issue in windows server 8’s new Powershell Web Access, I had the need to disable the windows firewall. Normally I would use a “netsh firewall” command, but when running this in windows server 8 we receive the notification:

“In future versions of Windows, Microsoft might remove the Netsh Functionality for windows firewall with Advanced Security”

“If you currently use Netsh firewall to configure and manage Windows Firewall with advanced Security, Microsoft recommends that you transition to Windows Powershell.”

This message continues in the screenshot below:





So with the warning in mind, I decided it was time to learn the new thing for the day and off I went to find out how to quickly disable the firewall using powershell!

As above the first clue is in the message, a new module seems to be available on both Windows 8 and Server 8. So with a quick get-module -listavailable I can see netsecurity listed:





“So what commands can I use with the netsecurity module” I asked myself. I ran a quick “Get-Command -module NetSecurity” and was overwhelmed with a large list of potential candidates as below:





So I used the trusted “Import-Module NetSecurity” command and set about reading the help files. Or so I thought…





As with PowerShell 3.0, the helpfiles for Powershell are not completely installed by default, we need to run update-help to download the powershell help files. This is causing quite a debate in the powershell community, so I’m not going to get into this, personally I think its a good idea.

Anyway, so off I went to update the help, or so I thought:





Bugger…

I’ll admit, this list had me scratching my head for some time, but I soon enough stumbled across the important command on this list for my objective. “Get-NetFirewallProfile”.

Get-NetFirewallProfile lists out the firewall profiles for your machine. The profiles are separated out into Domain, Public and Private as you can see in the screen-shot below:





Now that we’re aware of where the profiles live, its just a simple task of piping the current profiles, into a “Set-NetFirewallProfile” command to turn the profiles off.

Below is a quick on-liner to disable the windows firewall in Server 8 or Windows 8 completely:



[sourcecode language=”powershell”]
Import-Module NetSecurity -ea Stop ; Get-NetFirewallProfile | Set-NetfirewallProfile -Enabled False
[/sourcecode]



And that’s it!





And once finished troubleshooting, you can turn it back on as below:



[sourcecode language=”powershell”]
Import-Module NetSecurity -ea Stop ; Get-NetFirewallProfile | Set-NetfirewallProfile -Enabled True
[/sourcecode]