As one who still remembers (and has the scars) from those dreaded tape restore days to off site locations. I remember spending 72 hours + trying to restore machines to different hardware, struggle with different tape drive manufacturers and trying to find the right tape!  I jumped ship from sys admin long ago and moved to EUC, avoiding the backup market like the plague in later years so I really was a fresh set of eyes to this age old conundrum.

Recently I stumbled across Altaro VM backup as part of my usual day to day readings and my interest was piqued. Withstanding that as a home lab owner and a hypervisor snob I really only run vSphere and Hyper-V was normally not something I enjoyed working with… I got talking to a nice gentleman from Altaro and I was invited to join the closed beta for vSphere.

For those unfamiliar, Altaro recently released Altaro VM backup v6 with added support for vSphere. With this release and support for my beloved vSphere, I spun up an instance of Altaro, ditched the manual and decided to “have a go” to see if I could do it.

Installation:

The installation was a no brainer, I simply downloaded the software, ran the installation and opened the console. It was quick enough to not go for a coffee, which is good or bad depending on your mood at that time of the day.

Using the console:

(dashboard overview)

The Console reminds me a lot of Nutanix Prism for it’s simplicity and pleasing design. It’s completely wizard driven and in 6-7 steps From plugging in my hypervisors (vSphere, stand alone ESXi hosts or Hyper-V), to setting a backup location (local disk, removable disk, Nas, network share, etc.), to configuring a schedule took just minutes.

The console also has a great feature where you can manage multiple connections from a single console without feeling like you’re losing functionality.

Using the product:

 (Seed to disk example)

Altaro have made this dead simple. You have two storage types of choice and you can use multiple repositories for each type. Offsite and Onsite. As mine was just a lab, onsite made sense, but the ability to ship stuff offsite when required sounds like a great option. When bandwidth is limited to offsite, you can even “seed to disk” as depicted above and move it manually off site to save precious bandwidth. Super clever technology!

As this is VM based, I expected some hang ups on restoring VM contents. But Altaro have included a number of granular restores such as file and mailbox restores directly in the console. Nice touch. Altaro VM backup can also leverage VSS for “Application consistent” backups, this I was not expecting from a VM backup product and is very clever for when you need to leverage Microsoft VSS technologies.

(Granular file restore wizard)

Once configured, I decided to test a VM backup, a granular file restore and a VM restore. Restorations are very flexible, choosing the target host, network connectivity and restored device name. Wonderful approach and thought behind this process, it did what I wanted it to do in seconds. Happy with my progress I added a few virtual machines, set the backup schedule and wandered off back to whatever I was doing before and left it alone for a few weeks.

Support when you need it:

Until one faithful day last weekend my Citrix XenDesktop delivery infrastructure went kaboom.  I was performing my XenDesktop 7.6 upgrades to feature pack 3 (as you do when you’ve a home lab and like that kind of thing) and my storefront infrastructure decided not to uninstall or install. It decide to break and I couldn’t fix it. With the built in verification process in the product, I could quickly check that the backups were “OK” but something else was amiss….

I’ll be honest, as a developer and consultant, I use my lab every day, for tinkering, testing, Microsoft Visio and other such activities. Not having my lab on Monday morning was not an option I wished to think about. In my mild panic I remembered I had meant to review a backup product, I wondered if it was still working?

To my joy, it was working! It had taken a backup of the environment just last night and I set about performing a restore, but I had an issue.

Those rage inducing days in the datacenter were back, I had a backup? but I couldn’t restore!

But this is where Altaro saved the day for me. Inside of their console there’s a live help button! Within minutes I was connected to a support agent and my restore was running happily (it was my fault, consequently).

In review:

It took me less than 10 minutes to start my first backup. It ran in the background for a month backing up my stuff with no issues and saved me from a very nasty bug in Citrix storefront allowing me to restore my VM’s in minutes (saving me hours of work rebuilding my site).

You can grab a free copy here: http://www.altaro.com/vm-backup/download.php for up to 2 VM’s and a functionality matrix is available here too: http://www.altaro.com/vm-backup/pricing.php.

Having reviewed the features and prices of Altaro, this is definitely a product to keep in the back of your mind for environments suitable. It’s great software backed by great people with a strong focus on simplicity and killer functionality.

• ThinIO facts and figures, Part 1: VDI and Ram caching.
• ThinIO facts and figures, Part 2: The Bootstorm chestnut.
• ThinIO facts and figures, Part 3: RDS and Ram caching.

In the final blog post in this series, we’re going to discuss storage design and a frequent problem face when sizing storage. Lets get right into it:

“Designing for average, is designing for failure”

Peak IOPS:1015, Average IOPS: 78

A frequent mistake I see customers and consultants alike make is taking an average of a sizing requirement and using that as a baseline for sizing environments.

Looking at the figures produced from our internal load tests, we saw just an average of 78 IOPS required on write from a vanilla server without ThinIO to provide storage from this XenApp server.

Now frequently, people will take a figure like that, throw in 25% for growth and bob’s your uncle, ‘order the hardware Mr SAN man’. When I have questioned them about that assumption, they’ll often respond “oh it will be a bit slow if theres contention but it’ll even itself out”.

Right? Wrong.

Things don’t go slow when they are over subscribed, they stop.

Don’t take my word for it! Lets do some simple theoretical math:

If you take a storage device and allocate 100 IOPS to this machine, what’s going to happen when a peak like 1000 IOPS is requested? A lot of queuing. 

In theory, keeping to the 100 IOPS figure, that 1 second burst IO is now taking over 10 seconds to satisfy (1000 / 10).

But it gets worse, all subsequent IO that is requested after that spike occurred is going to also be haulted waiting for this task to occur.

Assuming you’re now mid spike and 10 seconds later the request is finished… taking your average figure, you now have 10 seconds worth of 100 IO’s per second potentially queued up behind…

Low and behold another login occurs and? STOP. Storage timeouts, twirly whirlies, application crashes, hour glasses and the good old “logins are slow”.

Oh ok, So how do I size to accommodate this? 

Well you’re between a rock and a hard place aren’t you. You can’t tell users when to login, the price tag of a SAN sized for peak activity + 20% is going to cost you more than your entire desktop estate. And as you can see, it’s never safe to assume it will run slow. 

Buying into shared storage is a tricky business 

Storage is expensive. Very expensive. It always annoys me when you hear vendors in this space refering to themselves as ‘reassuringly expensive’. To me this directly translates to ‘We can charge what we want, so we will and you can be reassured we feel the price tag is worth it.’

Storage was never written with desktop workloads in mind, it was written for ‘steady state’ server workloads and was in the progress of going the way of the ‘dodo’ (extinct) up until that first release of vMotion requiring shared storage, which some say was the saving of the market.

Many vendors are going with software or hardware intelligent tiering. This is a great feature, but the real question to ask is how frequently data is moved from the hot tier to lower tier? Press your vendor on this as they more than likely wont know! Microsoft storage spaces is a prime example of this with a really poor optimisation process of just once a day!

Then ask yourself what happens when a base image update occurs and changes the disk layout of the base golden image? Further, stateless technologies from the bigger vendors delete the differencing disk on restart, can you be sure the new disk is going to end up in the smaller, faster SSD or RAM tier? Or is data up there already in contention?

RAM is far less tricky

RAM is commodity, available in abundance and throughout every virtual desktop project I’ve architected and deployed, you run out of CPU resources in a ‘fully loaded’ host way before you will run out of RAM. RAM has no running maintenance cost, Ram is an upfront CAPEX cost and requires little to no maintenance.

The beauty of what ThinIO does with the little resources you assign it, is turn that desktop workload into a healthier and happier server workload of minimal burst IO and a low steady state IO requirement.

Note the peak of just 40.5 IOPS and average IOPS of less than 2.

with as little as just 200mb cache for each of the 10 users logging in, within an aggressive 3 minute window, we reduced the peak from 1000 to 40. That’s a 96% reduction in burst IO.

With ThinIO, you:

• reduce your exposure to massive IO spikes.
• Improve user logon times.
• significantly reduce your daily IOPS run rate.
• Increase user productivity by spending less time waiting for the storage.
• Commit to nothing up front, test it and see how well it works. If you are happy, then buy in.

Lots of Intelligence baked in:

ThinIO is acutely aware of key operating system events that will cause these kind of spikes and react accordingly to reduce the spikes in IOPS created. ThinIO constantly watches the behavior and IO pattern of the storage and tunes itself accordingly.

Unlike other technologies, ThinIO is a true caching and performance solution. We do not move useful data in and out of the cache on demand when cache availability is contencious. We track patterns and frequency of block access to respond accordingly, delivering all the benefits we have mentioned, even with the tiniest cache, without EVER reducing the capability of the storage when overwhelmed.

And on the opposite side of the scale, when underworked, we leverage our cache to deliver deeper read savings as above.

ThinIO also has a powerful API and PowerShell interface to allow you to report and interact with the cache on demand.

Wrap up:

And with the end of the series looming, allow me to finish on some easy points:

 ThinIO Allows you to:

• size your SAN outside of the Lamborghini category & price tag for your desktop estate.
• rapidly achieve far deeper density on your current hardware when you are feeling the Pinch.
• guarantee a level of performance by assigning cache per VM, disallowing other users to steal or hamper caching resources.
• Improve user experience and login times immediately.
• Reduce the impact of boot storms and similar IO storm scenarios.

No other vendor can offer as quick a turn around time with their product. ThinIO installs in seconds and offers a huge range of compatibility.

One more thing:

In case you missed ThinIO’s launch day at E2EVC Barcelona, ThinIO is now in GA, available from our website and production ready! More marketing to follow! But grab your copy now and get playing!

• ThinIO facts and figures, Part 1: VDI and Ram caching.

Getting right to it:

In this industry when somebody says ‘boot storms!’ – most of us will respond with:

Boot storms are a well documented, boring problem and have many solutions available from vendors and hypervisors alike. Most solutions today rely on a ‘shared memory’ storage area to cache ‘on boot’, in theory caching only one startup or one pattern in order to then serve it back to the proceeding desktops to boot.

But why are boot storms an issue? While working on ThinIO we had the unique ability to really dive into the Windows boot process and analyse why boot storms cause the damage they do and in this post we thought we’d share our findings to better document the issue.

Boot data:

Taking a typical windows 7 boot, to the login screen and idling until all services have started, the data traversing from disk to VM is relatively small. in our testing we found an average of Just 500-600 mb of data is read during this process, and write data barely registers at between 20 and 30mb.

But hey, what gives? Taking such low data throughput, why is boot such a contenscious issue? Have I been misled with marketing and vendor nonsense?

The IO chestnut:

Sadly no, it’s the way windows requests this data, but don’t take my word for it…. Behold, the incredible mess that is the Windows boot process!

Yep, that’s right, in the time Windows requested roughly 600mb of data, it sent down an astounding 70 thousand IO’s in the space of 2-3 minutes!

Math time:

Now if you were to take these figures as they stand, you would take 70,000 IO’s divide this into 560mb and you’d probably end up with an average of about 8k of data requested per IO… You’d be wrong.

As my good buddy Conor Scolard would say, ‘when you Assume, you make an ass out of you and me’.

To better understand the bounderies of Windows, Windows requests IO’s between the minimum of 512 bytes all the way up the spectrum later in the boot process to 128k and above. But it requests these blocks sparcely, on demand, and not just once per sector, the same blocks are frequently accessed.

The net result of this causes absolute havok on the storage:

The crux of the issue is, for each one of these IO’s, the storage provider needs to compute the block data requested, seek the data out, then return it.

But 70,000 of these IO operations for a meagre 600mb of data is madness and you can now see exactly why boot storms were labelled as such for those early adopters who had their hands burned by this fact finding mission.

I’ll mitigate this issue by just booting my VM’s at night!

I’m sure you will! I would also love to see your face if a number of users happen to restart their desktops during the day, cascading 70,000 IO’s per desktop to the storage in a 2 minute window, per desktop!

Bootstorming IS an issue.

Now, knowing all this, it makes sense as to why storage and hypervisors alike are using a cache of ram.

But how does ThinIO fit in here? With Read Ahead of course!

Knowing the Windows boot process as intimate as only a technology like ThinIO can, there are many, many optimisations we can make to this process.

We can both speed the boot process up and also massively reduce the storage requirement while in VM, without any fancy caching mechanism!

With ThinIO’s read ahead technology, we can deliver just shy of an 80% boot IO reduction with nothing other than having our technology in the virtual machine:

Taking a ThinIO averaged test and overlaying it to a baseline averaged test, it’s clear just how much impact this technology can have:

Wrap up:

So there you have it, with ThinIO, a simple, in VM solution, not only can you seriously reduce your IO footprint, boost user performance and achieve greater storage density per virtual machine, you also can also massively negate the impact a booting VM has on your storage.

If you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

Having been in this situation recently, I needed to audit and report on the types of files open on the file server, my hunch was a certain select number of users were running applications (like *gulp* lotus notes) from the network share.

Disappointed with the powershell scripts on the interwebs, I decided to write my own function to perform this task:

[sourcecode language=”powershell”]
function get-openfiles{
param(
$computername=@($env:computername),
$verbose=$false)
$collection = @()
foreach ($computer in $computername){
$netfile = [ADSI]"WinNT://$computer/LanmanServer"

$netfile.Invoke("Resources") | foreach {
try{
$collection += New-Object PsObject -Property @{
Id = $_.GetType().InvokeMember("Name", ‘GetProperty’, $null, $_, $null)
itemPath = $_.GetType().InvokeMember("Path", ‘GetProperty’, $null, $_, $null)
UserName = $_.GetType().InvokeMember("User", ‘GetProperty’, $null, $_, $null)
LockCount = $_.GetType().InvokeMember("LockCount", ‘GetProperty’, $null, $_, $null)
Server = $computer
}
}
catch{
if ($verbose){write-warning $error[0]}
}
}
}
Return $collection
}
[/sourcecode]

The function above (get-openfiles) has been written to accept an array of servers to the command line and it will return the following items:

• The ID of the open file.

• The server it’s open from.

• The username who has the file open.

• The amount of locks the file has.

A couple of quick examples for using this command are below:


Retrieving open files from server1:






[sourcecode language=”powershell”]get-openfiles -computername server1 | select server,itempath,lockcount[/sourcecode]



Retrieve a count of open files that end with the nsf file type (Lotus Notes):






[sourcecode language=”powershell”](get-open files -computername server1,server2 | ? {$_.itempath -like "*.nsf*"}).count()[/sourcecode]



Retrieve a report of total open files on a number of file servers:



[sourcecode language=”powershell”]get-openfiles -computername server1,server2,server3,server4,server5 | group -property server[/sourcecode]

“Your XenApp servers have very high disk queue’s and IO”

“What’s causing it?”

“dunno…”

With Server 2008, the task manager’s resource monitor feature will help you find these items. But in server 2003 this was a perilous task. The specific details for disk io per process are stored in performance monitor under each specific process running. Trying to analyse each process was a massive pain, but powershell can do some very clever work to help alleviate this!

I wrote two quick functions which act similar to “top” in linux for giving an on screen view, updating at interval of what exactly is creating IO activity. These two functions are:

get-IODataBytes:

Get-IODataOperations

The code for these functions are below:

[sourcecode language=”powershell”]
function get-iodatabytes{
$result=(get-counter -counter "Process(*)IO Data Bytes/sec" -ea 0).countersamples | ? {$_.cookedvalue -gt 0} | select instancename,@{Name="SessionID";Expression={if ($_.path.contains("#")){($_.path.split("#)"))[1]}else{"0"}}},@{Name="IO Data Bytes/sec";Expression={[math]::Round($_.cookedvalue,0)}},@{Name="IO Data KBytes/sec";Expression={[math]::Round($_.cookedvalue / 1024,0)}} | sort -Descending "IO Data Bytes/sec" | ft
$currentqueue=(((get-counter -counter "PhysicalDisk(0 C:)Current Disk Queue Length" -ea 0).countersamples) | select cookedvalue).cookedvalue
clear
write-warning "Hit [CTRL] + [C] to exit live capture"
write-host "Current Disk queue: $currentqueue"
return $Result
}

FUnction get-IODataOperations {
$result=(get-counter -counter "Process(*)IO Data Operations/sec" -ea 0).countersamples | ? {$_.cookedvalue -gt 0} | select instancename,@{Name="SessionID";Expression={if ($_.path.contains("#")){($_.path.split("#)"))[1]}else{"0"}}},@{Name="IO Data Operations/sec";Expression={[math]::Round($_.cookedvalue,0)}} | sort -Descending "IO Data Operations/sec" | ft
$currentqueue=(((get-counter -counter "PhysicalDisk(0 C:)Current Disk Queue Length" -ea 0).countersamples) | select cookedvalue).cookedvalue
clear
write-warning "Hit [CTRL] + [C] to exit live capture"
write-host "Current Disk queue: $currentqueue"
return $Result
}

[/sourcecode]

if you wish to loop one of these functions, simply use the following code:

[sourcecode language=”powershell”]
while ($true){
get-iodataoperations
start-sleep 1
}
[/sourcecode]

Hello there, and thank you for dropping back for part 3…

I suppose I should start with the disappointing news that I have yet to test this option for VDI in a box. And despite Aaron Parker’s suggestions it wasn’t due to lack of inspiration, it was down to lack of time! This series has gathered allot of interest from both community and storage vendors alike, and I feel I should set the record straight before I got any further:

1. This isn’t a production idea, you would be crazy to use this idea in a live environment.
2. Throughout this entire project, we’re focusing on pooled stateless. Stateful desktops would be a separate post entirely.
3. This wasn’t an attack on products in this market space, merely a fresh view on an old problem.
4. If i had the skills or funds necessary to run this project to a production solution, I wouldn’t have posted it. I would already be hard at work creating a reasonably priced product!

Now that my declarations are out of the way, I’d first like to talk about the moral of the story. This isn’t an unfamiliar expression:

IOPS mitigation is not about read IOPS it’s about WRITE IOPS!

VMware, Citrix and Microsoft have similar but very different solutions for read IOPS negotiation. Similar in the sense that they try to negate storage read IOPS. But the key difference with XenServer is the local disk cache via Intellicache has the out of box functionality to cache the majority of read’s to local disk (think SSD*) without the baked in soft limit of 512 MB for Microsoft HyperV and VMware respectively.

Long story short, VMware and Microsoft’s solution is about 512mb of recognizable read IOPS negation un-tuned, but enabled. Of course this value can be tuned upwards, but the low entry point of cache would suggest, at least to me, that tuning up will have an upsetting affect on the host.

This to me is why IntelliCache has the upperhand in the (value add product) VDI space for read IOPS negation and they even throw in the Hypervisor as part of your XenDesktop licensing, so win win, but what about those pesky write IOPS?

Let’s look at Citrix for a moment!

If we review article 1 and article 2 for a moment, the half baked idea formed by Barry Schiffer, Ingmar Verheij, Kees Baggerman, Remko Weijnen and I, when combined with Citrix IntelliCache turned out to be a phenomenal combination to create an IOPS avoidance product. But the key point we had in the back of our heads was, Citrix provisioning server already has this driver!

Citrix provisioning server has a RAM cache driver, with configurable size baked into the Provisioning Services product. This driver works in a very similar way to the EWF driver, just without the API and flexibility of a Microsoft driver.

There are customers of Citrix out there using RAM caching with XenApp who I spoke to, they have assigned a large (8gb+ cache, 3-4gb of cache utilised day to day) to negotiate the chance of a spillover, but it could still happen and this is an acceptable risk to their implementation. I struggled to find a XenDesktop customer using this method who would talk about their implementation.

But with XenDesktop in mind, using an overly large cache size to avoid spillover just doesn’t scale the way a consolidated Server Based Computing model would… Which leaves us back in this same dilemma, I’d like to cache in RAM, but I don’t want to risk users losing data when they perform a task we have not factored for.

So with all this in mind, lets talk about my final hurdle I faced with this project.

RAM cache flood and my half baked solution:

The Microsoft EWF driver suffered the same problem as the Citrix provisioning server when the cache filled, i.e. it froze or outright bug checked if you tried to reference a file put into the cache before you filled it with other information.

Jeff Silverman has done a great article on what happens in this event with the Citrix Provisioning Services RAM cache and I urge you to go read it so I don’t have to repeat it! Go now, I’ll wait right here for you to get back!

Read that article? Ok good, let’s continue.

To combat this scenario with the Windows EWF driver, I created a very rough around the edges windows service using polling (Sorry Remko) to check the size of the cache periodically via the EWFAPI and write the cache out to disk.

The function in the EWFAPI called EwfCommitAndDisableLive allowed me to dump the ram cache to disk and subvert the cache, going straight to disk when this event had occurred. By using this routine the ram cache simply disables itself and allows pass-through to disk at this point.

With this in mind, I tuned my service to spill to disk when the cache was greater than 80% of ram available, not in use by the operating system when the system booted, This worked well to a point, but the key failure to this approach became apparent when you opened a large number of applications and they struggled to occupy the space available around the cache.

My second attempt however, proved much more fruitful where I monitored for free system bytes in memory and if this dropped below a certain threshold the EWF drive began it’s dump routine to disk. Once the dump routine was complete, ram cleared and the writes had been committed to storage where the disk continued to use it for the remainder of the session. Once this spill over had occurred, a notification bubble fired in the user session warning them of the degradation of service and they should log off saving work at the next convenient moment…

Voila! no blue screen, spill to disk and user notification of degradation.

This wasn’t fool proof, it was very rough, it didn’t handle files being written larger than the RAM cache, but In my opinion, it satisfied the biggest fear and business case against using the Citrix Provisioning server ram caching, the ram cache flood scenario. I was happy with the results and it scaled well to the size of my lab, it showed what is possible with a RAM filter driver and it allowed me to prove my point before I started to poke the hornets nest of the storage market. So let’s park the EWF story for now and focus on my favorite subject, Citrix technologies.

Note: I won’t be making this service and driver solution publicly available, it’s not a production ready solution and Microsoft would sue the pants off me. I’m sure you’ll understand why, but if you want more information drop me an email.

The next part’s of this blog are all theoretical, but I know certain people I want to listen, are listening (Hi Kenneth, hope you are well :)).

Negating the negated. Lets talk about that spill over.

But what about that Spill over event? by using a spill over from ram to disk, we create a situation where we could change a steady, slightly reliable IOPS per from each desktop, to a situation where, “couldn’t a collection of spillovers at the same time cause your storage to becoming swamped with IOPS?”

Absolutely, and I’ve been thinking about this quite a bit… But there’s another hidden potential here even in a RAM spill over scenario…

With a little bit of trickery couldn’t we also negate this spillover event with a simple provisioning job?

With a bit of time spent thinking about this scenario, this is what I came up with…

Why doesn’t the controller, for XenApp or XenDesktop (PVS / DDC) copy or (my personal preference) create a local, blank differencing disk when a VM boots?

The hypervisor could be completely agnostic at this point, we could spill over to local disk, keeping write IOPS completely away from the shared storage? even in a spill over event?

This approach (in this half baked enthusiasts view) would negate the negated… But don’t take my word for it, lets go for some theoretical solutions.

Solution A: Implementing a spill over routine with the Citrix provisioning server driver.

So with my success with the Microsoft driver, I got myself thinking how easy would this be to do utilizing the Citrix Provisioning Services driver? Without access to the code, I’m going to be making slightly risky statements here, but I have allot of faith in Citrix that they could make this happen.

From everyone I have spoken to about this idea, they all see the value in the ability to spill out of RAM… So Citrix, please make it so. Below are some idea’s for deployment methods assuming Citrix do march down this route and the pro’s and con’s I see that live with each scenario.

Bear in mind, I’m not a storage guy, a full time developer or a software architect, I’m just an enthusiast that see’s potential, so drop your comments below as to what you think!

Machine Creation Services.

Idea A: Provisioning services improved RAM caching, Machine Creation services and Intellicache on XenServer.

Utilizing XenServer and MCS we could leverage the intellicache to negate the read IOPS as we proved in my own EWF driver testing, but still deliver on that spill over mechanism allowing continuation of service.

Pros:

• Read IOPS negation.
• RAM caching to remove write IOPS (bar a spill over).
• Continuation of service in a cache flood scenario.

Cons:

• Limited to XenServer.
• Over provisioning of RAM necessary per desktop.
• RAM spillover will result in a large amount of IOPS to the shared storage.

Idea B: Provisioning services improved RAM caching, Machine Creation services and Intellicache on XenServer… With local copy!

Same benefits as previous, but now, we have zero reliance on the shared storage when the VM is up (except for ID disk actions).

Pros:

• Read IOPS negation.
• RAM caching to remove write IOPS.
• Uses local resources in a spill over.
• Continuation of service in a cache flood scenario.

Cons:

• Limited to XenServer.
• Over provisioning of RAM necessary per desktop.

So that’s MCS in a nutshell with per VM caching, I think this solution has so much potential I can’t believe it’s not been done, but I digress. So lets park that topic for now and move on to Citrix Provisioning Services.

Citrix Provisioning Services:

So lets look at the “favorite of many” technology.

Idea A: Citrix Provisioning Services and Improved RAM caching driver.

In a pure Provisioning services environment, we would force our read IOPS via the lan, instead of storage protocol but still deliver a spill back to disk to allow continuation of service.

Pros:

• Hypervisor agnostic.
• RAM caching to remove write IOPS (bar a spill over).
• Continuation of service in a cache flood scenario.
• Potentially no shared storage needed, at all, if caching on the PVS server.

Cons:

• Read IOPS aren’t really negated, they’re just forced over another technology.
• Over provisioning of RAM necessary per desktop.
• RAM spillover will result in a large amount of IOPS to the shared storage / pvs server.

Idea B: Citrix Provisioning Services and Improved RAM caching driver.. With local copy!

Taking the above benefits, but with the gain of utilizing local storage in the spillover event.

Pros:

• Hypervisor agnostic.
• RAM caching to remove write IOPS.
• Uses local resources in a spill over.
• Continuation of service in a cache flood scenario.
• Potentially no shared storage needed, at all.

Cons:

• Read IOPS aren’t really negated, they’re just forced over another technology.
• Over provisioning of RAM necessary per desktop.

So lets Review:

And there we have it, 4 solutions to IOPS negation utilizing the Provisioning server RAM caching driver, with a little bit of a modification to deliver a robust solution to RAM caching.

The copy and creation of differencing disks again would deliver additional benefits to leverage the hardware you put into each Hypervisor without the Shared Storage investment.

Win Win, but is it?

There’s an oversight here:

There’s a niggle here that’s been bothering me for some time and you’ll probably note I mentioned it as a CON to most of the solutions above… I’m going to lay it out on the table in complete honesty…

“Isn’t over provisioning RAM on a per desktop basis a waste of resource? Wouldn’t it be better if we could share that resource across all VM’s on a Hypervisor basis?”

You betcha! If we are assigning out (for argument sake) 1gb of ram cache per VM, that RAM is locked into that VM and if another machine spills, the free RAM in other desktops is wasted.

You would be completely insane not to reserve this RAM per VM, if an over-commit for the VM’s is reached this RAM will merely spill out to a page file type arrangement, negating all your benefits.

Ultimately, assigning RAM in this way could be seen as wasteful in the grand scheme of things…

But there are other ways to skin this cat!

So this leads me on to something I was also considering which popped up in a twitter conversation recently. RAM disk’s and Hypervisors.

Current storage vendors will create a storage pool, consisting of ram inside a VM, per hosting hypervisor for local storage to allow good IOPS leveraging that VM. This VM can perform additional RAM optimizations like compression, de-duplication and sharing of similar pages to reduce the count.

This approach is super clever, but, In my humble opinion (please don’t kill me vendors), this approach is wasteful. Running a VM as a storage repository per host has an overhead to run this virtual machine and takes away from the agnostic solution that is possible…

What If the hypervisor provides the RAM disk?

So excluding ESXi for a second, as getting a RAM disk into that platform would require VMware to allow access to the stack. Lets look at Hyper-V (2012) and XenServer for a second…

With Unix and Windows platforms, RAM disks have been available for years. They were once a necessity and a number of vendors still provide them for high performance IO environments.

Lets say (for arguments sake) Citrix and Microsoft decide to provide a snap-in to their hypervisor to allow native RAM disks (or a vendor writes one themselves!) and maybe, they even decide to provide RAM compression, Spill over to local disk, dedupe, and page sharing on this volume from the hypervisor stack…

Wouldn’t this extend provide all the benefits we’ve spoken about, without the need for a VM per host? And using Thin Provisioning allow all desktops to share the large pool of RAMdisk available?

Yes, yes, it would.

Above is an example of how I would see this working.

So random pictures are fine and all, but what about the read IOPS negation technologies? and what about combining these with XenServer or Hyper-V?

XenServer and IntelliCache:

Well there you have it now, all the benefits of a per VM filter, leveraging intellicache for reads and spilling out to local disk… RESULT!

Pro’s:

• Read IOPS negation
• Write IOPS negation
• No Shared Storage required for running VM’s
• Shared pool of RAM for all desktop’s to use.

Con’s:

• A small one, but no migration of VM’s.

Provisioning server?

and again, all the benefits of a per VM filter, reads redirected via lan and spilling out to local disk… RESULT!

Pro’s:

• Write IOPS negation
• No Shared Storage required for running VM’s
• Shared pool of RAM for all desktop’s to use.

Con’s:

• A small one, but no migration of VM’s.
• No real read IOPS negation.

And HyperV + CSV Cache?

Well here’s an accumulation of my thoughts:

So lets just talk for a second about what we are seeing… Utilizing a RAM disk with spill over, and copied vhd’s on boot we are removing the need for shared storage completely and hosting the IOPS from the hypervisor, natively without the need for additional VM’s.

And see that little icon down the bottom? Yep, that’s right, live migration from host to host thanks to Microsofts Shared Nothing Live Migration!

Pro’s:

• Some write IOPS negation
• No Shared Storage required for running VM’s
• Liv migration.
• Shared pool of RAM for all desktop’s to use.
• write back to local disk.

Con’s:

• No full read IOPS negation.

Review.

I’m sure there’s loads of reading in here, and there will be tons of thought and questions after this blog post. This has been in my head for roughly 9 months now and it feels victorious to finally get it all down on paper.

At the end of the day, Ram Caching is the way of the future, you pay no maintenance on it, you keep your IOPS off your storage and with a little magic from Microsoft, or particularly Citrix. You could see these benefits.

If / When these technologies make it out to you, you could quite happily stick with Machine Creation services, leverage intellicache and your shared storage requirement could be a cheap NAS from any vendor. Provisioning services also see’s allot of benefits from this approach, but the real creme de la creme is in that intellicache feature.

The key functionality to these approaches is simple:

• Cache RAM
• Spill to disk
• Keep the differences on local storage if it spills.

One last thing?

In true Mark Templeton fashion, you know I have a one last thing. This actually hit me today while writing this post and to be honest, I’m so amazed by the potential of this idea I’m going to build it in the lab this weekend.

But until then, a teaser…

What if I told you there was already an overlay, freely available baked into a Microsoft technology that would allow you to overlay local storage in Hyper-V with RAM. A powerful API attached to it, and the ability to write specific files to the local disk, subverting the overlay only when needed?

• RAM disk type technology? > yep
• already available? > yep
• Powerful API? > yep
• already can spill to disk? > yep

Yep Yep Yep! check back in a few days.

(If you know what it is already, keep it to yourself until I’ve gotten out of the lab!)

What do you think?

It doesn’t take a half baked enthusiast like me to see this potential and I’d be really eager to hear your comments on these approaches. If you would prefer to keep your comments offline, you can reach me on andrew@andrewmorgan.ie.

Acquiring the Codec’s:

1: Head over to the microsoft download site and download the Windows Embedded Standard 7 Service Pack 1 Tookit (part’s 1 to 8)
2: Once fully downloaded, run the Standard 7 SP1 Toolkit.part01.exe and extract the contents to a folder (e.g. c:toolkit)
3: Once fully extracted, browse to C:toolkitStandard 7 SP1 ToolkitDSPackagesFeaturePack (where the extract folder was c:toolkit).
4: In this folder, you will find the codec’s stored in folders beginning with “x86~winemb-premiumcodecs

5: Make a new folder in your c: drive called codecs, and copy the contents of the above folders into this new folder. The folder should appear as below:

6: Once we have the codec’s we can now integrate them using two methods, we can add them to a current machine, or add them to an installer image. Both Method’s are included below:

Online Method:

1: From a Windows Thin PC, run the following command from an administrative command prompt:

Dism.exe /Online /Add-Package /PackagePath:C:codecs /NoRestart

Integrating the Codec’s into an image:

1: Extract your ISO copy of the Windows Thin PC to c:windows_thin_pc

2: create a new folder to mount the image to, called c:mnt

3: Run the following command from an administrative command prompt:

Dism.exe /Mount-Wim /WimFile:C:Windows_Thin_PCsourcesinstall.wim /index:1 /MountDir:c:Mnt
Dism /Image:c:Mnt /Add-Package /PackagePath:c:codecs
Dism.exe /Unmount-Wim /MountDir:c:Mnt /commit

4: Once this completes, add your image back to Microsoft WDS, or write it to a usb key or DVD.

Note: if the command prompt reports DISM.exe is unavailable, grab a copy of the Microsoft WAIK.

To alleviate the issue, I added a simple shell extension to perform unattended installs of each of the msi’s:

I’ve made the registry file available here for download should you also be stuck with this task at some point.

The registry file is very simple as you can see below:

Had a requirement recently to disable flash on a user by user basis in our published desktop environment, this key wasnt readily available on the internet so here it is for anyone else wishing to do the same:

[HKEY_CURRENT_USERSoftwareMicrosoftWindowsCurrentVersionExtSettings{D27CDB6E-AE6D-11CF-96B8-444553540000}]
"Flags"=dword:00000001
"Version"="*"

when right clicking  on an office document Adobe decided  “Combine supported files in acrobat” to be we all must have, fine in the wild… but not so good in a shared desktop environment.

To remove this from your Citrix environment / Desktop simply run this command:

REGSVR32 /u "C:Program FilesAdobeAcrobat 9.0Acrobat ElementsContextMenu.dll.

If you also wish to remove the New > file type associations that Adobe decide to push in, heres a script to do so:

REG DELETE HKEY_CLASSES_ROOT.xdpAcroExch.XDPDocShellNew /f
 REG DELETE HKEY_CLASSES_ROOT.xdpShellNew /f

Update:

With the move to 64 bit platforms, a new dll has been put in place:

C:Program Files (x86)AdobeAcrobat 9.0Acrobat Elementscontextmenu64.dll

This dll add’s the following items to the context menu:

• Convert to Adobe PDF
• Convert to Adobe PDF and Email.

If you wish to remove these, run the following command:

REGSVR32 /u "C:Program Files (x86)AdobeAcrobat 9.0Acrobat ElementsContextMenu64.dll"