• ThinIO facts and figures, Part 1: VDI and Ram caching.
• ThinIO facts and figures, Part 2: The Bootstorm chestnut.
• ThinIO facts and figures, Part 3: RDS and Ram caching.

In the final blog post in this series, we’re going to discuss storage design and a frequent problem face when sizing storage. Lets get right into it:

“Designing for average, is designing for failure”

Peak IOPS:1015, Average IOPS: 78

A frequent mistake I see customers and consultants alike make is taking an average of a sizing requirement and using that as a baseline for sizing environments.

Looking at the figures produced from our internal load tests, we saw just an average of 78 IOPS required on write from a vanilla server without ThinIO to provide storage from this XenApp server.

Now frequently, people will take a figure like that, throw in 25% for growth and bob’s your uncle, ‘order the hardware Mr SAN man’. When I have questioned them about that assumption, they’ll often respond “oh it will be a bit slow if theres contention but it’ll even itself out”.

Right? Wrong.

Things don’t go slow when they are over subscribed, they stop.

Don’t take my word for it! Lets do some simple theoretical math:

If you take a storage device and allocate 100 IOPS to this machine, what’s going to happen when a peak like 1000 IOPS is requested? A lot of queuing. 

In theory, keeping to the 100 IOPS figure, that 1 second burst IO is now taking over 10 seconds to satisfy (1000 / 10).

But it gets worse, all subsequent IO that is requested after that spike occurred is going to also be haulted waiting for this task to occur.

Assuming you’re now mid spike and 10 seconds later the request is finished… taking your average figure, you now have 10 seconds worth of 100 IO’s per second potentially queued up behind…

Low and behold another login occurs and? STOP. Storage timeouts, twirly whirlies, application crashes, hour glasses and the good old “logins are slow”.

Oh ok, So how do I size to accommodate this? 

Well you’re between a rock and a hard place aren’t you. You can’t tell users when to login, the price tag of a SAN sized for peak activity + 20% is going to cost you more than your entire desktop estate. And as you can see, it’s never safe to assume it will run slow. 

Buying into shared storage is a tricky business 

Storage is expensive. Very expensive. It always annoys me when you hear vendors in this space refering to themselves as ‘reassuringly expensive’. To me this directly translates to ‘We can charge what we want, so we will and you can be reassured we feel the price tag is worth it.’

Storage was never written with desktop workloads in mind, it was written for ‘steady state’ server workloads and was in the progress of going the way of the ‘dodo’ (extinct) up until that first release of vMotion requiring shared storage, which some say was the saving of the market.

Many vendors are going with software or hardware intelligent tiering. This is a great feature, but the real question to ask is how frequently data is moved from the hot tier to lower tier? Press your vendor on this as they more than likely wont know! Microsoft storage spaces is a prime example of this with a really poor optimisation process of just once a day!

Then ask yourself what happens when a base image update occurs and changes the disk layout of the base golden image? Further, stateless technologies from the bigger vendors delete the differencing disk on restart, can you be sure the new disk is going to end up in the smaller, faster SSD or RAM tier? Or is data up there already in contention?

RAM is far less tricky

RAM is commodity, available in abundance and throughout every virtual desktop project I’ve architected and deployed, you run out of CPU resources in a ‘fully loaded’ host way before you will run out of RAM. RAM has no running maintenance cost, Ram is an upfront CAPEX cost and requires little to no maintenance.

The beauty of what ThinIO does with the little resources you assign it, is turn that desktop workload into a healthier and happier server workload of minimal burst IO and a low steady state IO requirement.

Note the peak of just 40.5 IOPS and average IOPS of less than 2.

with as little as just 200mb cache for each of the 10 users logging in, within an aggressive 3 minute window, we reduced the peak from 1000 to 40. That’s a 96% reduction in burst IO.

With ThinIO, you:

• reduce your exposure to massive IO spikes.
• Improve user logon times.
• significantly reduce your daily IOPS run rate.
• Increase user productivity by spending less time waiting for the storage.
• Commit to nothing up front, test it and see how well it works. If you are happy, then buy in.

Lots of Intelligence baked in:

ThinIO is acutely aware of key operating system events that will cause these kind of spikes and react accordingly to reduce the spikes in IOPS created. ThinIO constantly watches the behavior and IO pattern of the storage and tunes itself accordingly.

Unlike other technologies, ThinIO is a true caching and performance solution. We do not move useful data in and out of the cache on demand when cache availability is contencious. We track patterns and frequency of block access to respond accordingly, delivering all the benefits we have mentioned, even with the tiniest cache, without EVER reducing the capability of the storage when overwhelmed.

And on the opposite side of the scale, when underworked, we leverage our cache to deliver deeper read savings as above.

ThinIO also has a powerful API and PowerShell interface to allow you to report and interact with the cache on demand.

Wrap up:

And with the end of the series looming, allow me to finish on some easy points:

 ThinIO Allows you to:

• size your SAN outside of the Lamborghini category & price tag for your desktop estate.
• rapidly achieve far deeper density on your current hardware when you are feeling the Pinch.
• guarantee a level of performance by assigning cache per VM, disallowing other users to steal or hamper caching resources.
• Improve user experience and login times immediately.
• Reduce the impact of boot storms and similar IO storm scenarios.

No other vendor can offer as quick a turn around time with their product. ThinIO installs in seconds and offers a huge range of compatibility.

One more thing:

In case you missed ThinIO’s launch day at E2EVC Barcelona, ThinIO is now in GA, available from our website and production ready! More marketing to follow! But grab your copy now and get playing!

To recap, below you will find the previous articles:

• ThinIO facts and figures, Part 1: VDI and Ram caching.
• ThinIO facts and figures, Part 2: The Bootstorm chestnut.

Off topic note:

two years ago at an E2EVC event, the concept behind ThinIO was born with just a mad scientist idea amongst peers.

If you are lucky enough to be attending E2EVC this weekend, David and I will be there presenting ThinIO and maybe, just maybe there will be an announcement. Our session is on Saturday at 15:30 so pop by, you won’t be disappointed.

Back on topic:

So here’s a really interesting blog post. Remote Desktop Services (XenApp / XenDesktop hosted shared) or whatever you like to call it. RDS really presents a fun caching platform for us, as it allows us to deal with a much higher IO volume and achieve deeper savings.

We’ve really tested the heck out of this platform for how we perform on Microsoft RDS, Horizon View RDS integration and Citrix XenSplitPersonality with Machine Creation Services.

The figures we are sharing today are based on the following configuration and load test:

• Citrix XenDesktop 7.6
• Windows Server 2012 r2
• Citrix User Profile Manager.
• 16gb of Ram.
• 4 vCpu.
• LoginVSI 4.1 medium workload 1 hour test.
• 10 users.
• VMFS 5 volume.

Fun figures!

Diving straight in, lets start by looking at the volume of savings across three cache types.

Reviewing the details for a moment:

Running repetitive tests of at least 3 per cache type, we found even at the lowest entry point we would support (50mb per user) we saw phenomenal savings of over 70% on write IO.

No pressure no diamonds!

To put that into perspective, at a 512 MB cache for 10 users, our cache reached maximum capacity at the second user login. With 8 users still left to login, cache full and still an hours worth of load testing left, our ThinIO technology was under serious pressure.

This is key to why ThinIO is such a great solution. We won’t just perform great until we fill our cache, we don’t require architecture changes or care about your storage type, we have no lead times or install days, we will carry on to work with what is available to use, to take a large ammount of pressure off storage IOPS and data throughput.

With the figures above, you can see just how well the intelligence behind our cache can scale even when it faces such a steep workload.

Below you will find a breakdown of each test:

512 MB cache:

Breaking down into the figures, on the 512mb cache test, it’s clear to see just how well ThinIO deals with the tiniest of caches:

When we side by side this with our baseline averages, you can see we take a huge chunk out of that Spiky login pattern and continue to  reduce the steady state IO as the test continues:

So lets move up and see how we get on!

1024 mb cache:

Doubling up our cache size we see a great increase in both read and write savings as you’d expect.

With 100mb of cache per user, and the average user profile in the test 3 times that size. We are still under pressure. As we will natively favour optimisations to write IO over read, you’ll see the bulk of improvements happen in write when we’re under pressure as illustrated in this test:

With more cache available during the peak IO point, we make further savings on write:

2048 mb cache:

and at our recommended value of 200mb per user in Remote Desktop Services, the results are phenomenal! With this size, even still below the 300mb mark per user profile, the read IO gets a really good boost and the write IO saving well over the 95% mark!

And the side by side comparison is every bit as good as the savings illustrated above, reducing that peak bursty IO to just 41 IOPS:

But there’s more! 

As i pointed out in the previous blog, IOPS are just one side of the story. A reduction of data throughput to the disk is also a big benefit when it comes to storage optimisation, and as you can see we make a big difference:

Wrap up:

So there you have it, with ThinIO, a simple, in VM solution, you can you seriously reduce your IO footprint, boost user performance and achieve greater storage density per virtual machine or on Remote Desktop Services technology.

In the mean time:

If you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

• ThinIO facts and figures, Part 1: VDI and Ram caching.

Getting right to it:

In this industry when somebody says ‘boot storms!’ – most of us will respond with:

Boot storms are a well documented, boring problem and have many solutions available from vendors and hypervisors alike. Most solutions today rely on a ‘shared memory’ storage area to cache ‘on boot’, in theory caching only one startup or one pattern in order to then serve it back to the proceeding desktops to boot.

But why are boot storms an issue? While working on ThinIO we had the unique ability to really dive into the Windows boot process and analyse why boot storms cause the damage they do and in this post we thought we’d share our findings to better document the issue.

Boot data:

Taking a typical windows 7 boot, to the login screen and idling until all services have started, the data traversing from disk to VM is relatively small. in our testing we found an average of Just 500-600 mb of data is read during this process, and write data barely registers at between 20 and 30mb.

But hey, what gives? Taking such low data throughput, why is boot such a contenscious issue? Have I been misled with marketing and vendor nonsense?

The IO chestnut:

Sadly no, it’s the way windows requests this data, but don’t take my word for it…. Behold, the incredible mess that is the Windows boot process!

Yep, that’s right, in the time Windows requested roughly 600mb of data, it sent down an astounding 70 thousand IO’s in the space of 2-3 minutes!

Math time:

Now if you were to take these figures as they stand, you would take 70,000 IO’s divide this into 560mb and you’d probably end up with an average of about 8k of data requested per IO… You’d be wrong.

As my good buddy Conor Scolard would say, ‘when you Assume, you make an ass out of you and me’.

To better understand the bounderies of Windows, Windows requests IO’s between the minimum of 512 bytes all the way up the spectrum later in the boot process to 128k and above. But it requests these blocks sparcely, on demand, and not just once per sector, the same blocks are frequently accessed.

The net result of this causes absolute havok on the storage:

The crux of the issue is, for each one of these IO’s, the storage provider needs to compute the block data requested, seek the data out, then return it.

But 70,000 of these IO operations for a meagre 600mb of data is madness and you can now see exactly why boot storms were labelled as such for those early adopters who had their hands burned by this fact finding mission.

I’ll mitigate this issue by just booting my VM’s at night!

I’m sure you will! I would also love to see your face if a number of users happen to restart their desktops during the day, cascading 70,000 IO’s per desktop to the storage in a 2 minute window, per desktop!

Bootstorming IS an issue.

Now, knowing all this, it makes sense as to why storage and hypervisors alike are using a cache of ram.

But how does ThinIO fit in here? With Read Ahead of course!

Knowing the Windows boot process as intimate as only a technology like ThinIO can, there are many, many optimisations we can make to this process.

We can both speed the boot process up and also massively reduce the storage requirement while in VM, without any fancy caching mechanism!

With ThinIO’s read ahead technology, we can deliver just shy of an 80% boot IO reduction with nothing other than having our technology in the virtual machine:

Taking a ThinIO averaged test and overlaying it to a baseline averaged test, it’s clear just how much impact this technology can have:

Wrap up:

So there you have it, with ThinIO, a simple, in VM solution, not only can you seriously reduce your IO footprint, boost user performance and achieve greater storage density per virtual machine, you also can also massively negate the impact a booting VM has on your storage.

If you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

Test, test, review and tune. Rinse and repeat!

We’ve spent months load testing, tuning, fixing and retesting ThinIO. And for the first time we’re going to start talking about the dramatic results ThinIO can have on storage scalability and user perceived performance.

During our extensive testing cycles, we’ve covered:

• Horizon View
• Citrix XenDesktop
• Microsoft RDS

We’ve been seeing very similar, if not identical results when testing against pools on the following storage types too:

• XenServer SR
• VMFS
• NFS
• Microsoft Clustered Shared Volumes

For reference the statistics we’re sharing today are based on VDI via VMware Horizon View 6, these figures are averages across at least three independent tests. All details of the tests that exported these results are covered below.

Testing details:

The VM’s we tested in this particular workload are as follows:

• Testing Method: Login VSI 4.1 Medium Workload
• Operating System: Windows 7 x64 SP1
• System ram: 3gb
• vCPU: 2
• ThinIO cache: 350mb
• Technology:  VMware Horizon View 6
• Test runtime: 1 hour*
• Statistic sample period: 5 seconds.

With that out of the way, lets jump right in!

Storage IO:

The number of IO’s per second is crucially important when dealing with storage; many, many small IO’s sent to sparse locations on disk are a killer to storage technologies, only made worse by certain file systems.

As a storage acceleration and negation technology, were extremely happy with the IO’s reduction we see on the storage:

Even with just 350 MB of ram as a cache, we achieve phenomenal IO reduction.

Storage MB / Sec:

But IO’s are just one part of the puzzle, what about the size of the data requests being sent to the storage?

A true solution to take the pressure off the SAN, improve user performance, and increase storage density needs to tackle both the IO and the throughput problem.

As you can see above, with just 350mb we’re very good at it!

Side by Side Comparison:

So rounded figures are fine so long as the data is trustworthy, but here’s a real preview laid bare for your analysis.

Here’s a direct comparison on NFS and VMFS of taking a standard load test IO pattern and comparing it to an identical test with ThinIO installed in the VM:

VMFS:

NFS:

As you can imagine, we’re extremely proud of what we can achieve with as little as 350mb per desktop.

The beauty of our approach is simplicity, your users can see this benefit not in a matter of weeks, days or even hours. ThinIO can be up and running in minutes, delivering reduced login times, storage acceleration and providing a far deeper density on your current storage.

Wrap up:

So there you have it, we’ll be adding additional blog posts in the coming days looking at Remote Desktop Services / XenApp, intelligent cache management built in and our Read Ahead technology, so check back. In the mean time, if you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

* Eight-hour figures and complete statistics are also available, we have nothing to hide and we’d encourage you to get in touch with the ThinScale team and we’ll share them with you.

Warm up your labs or fire up your golden images ladies and gents, we’re delighted to announce ThinIO’s brief public beta will begin today!

This project has taught us some really interesting things about Windows IO, how Windows behaves and how the hypervisor and storage can behave. This project really felt like a David vs. Goliath task as we (members of our community with a desire to simplify this issue) attempted to tackle one of the largest issues in our industry, storage bottlenecks and Windows desktops.

What’s really unique about our approach is there are no hardware lead times, no architecture changes needed and no external dependencies. ThinIO can be installed in seconds and the benefits are seen immediately.

We’ve spent countless hours testing, tuning, retesting and even more tuning. We’re extremely happy with the results. This public beta will serve as an opportunity for you to really kick the tyres and believe the hype in what we’ve built while we’re putting together the final touches to release the product in the coming weeks.

During this time, we found achieving positive and consistent IO negation boils down to a number of items:

• cutting down on the volume of IOPS sent to the storage.
• Reducing the data transferred (MB/sec) to and from the storage.
• Intelligently cutting down on peak IO, such as boot and user logon.

In the coming days we’re going drill down into these categories in more depth. But as a quick overview, here’s a baseline (top) and ThinIO (bottom) session comparison of a Windows 8.1 desktop login, 1 hour Login VSI medium workload and log off with just 350 mb of cache for ThinIO:

Keep an eye out for the coming blog posts, but in the mean time, the ThinIO beta is available to download here now! Go forth and have fun.

Until next time,

A

For those unfortunate enough to miss synergy or our Webinar with Erik over at XenAppBlog, here’s a little blog post you will find interesting as I walk you through the inners of ThinIO and why it’s so simple to deliver disk access with RAM speeds without any of the complexity.

What is ThinIO?

ThinIO is a filter driver which operates at a block level, inline between the windows and the disk.

ThinIO sits in the operating system layer and can be used on windows desktop operating systems or server based computing models.

ThinIO delivers a greatly reduced IO footprint on your storage, while also speeding up core items like boot times and login times. ThinIO also helps standardize the peaks your storage will get hit by at busy periods during the day. Ultimately this allows you to size your storage for an average, as opposed to sizing for the worst case scenario during peaks.

How does it work?

When ThinIO starts up, it allocates a configurable cache of reserved RAM to perform its optimisations.

Being the last filter in the stack, ThinIO can still allow windows to perform it’s own optimisation on IO, delivering value by catching the read and write IO’s just as they hit the disk.

ThinIO interacts with block data as read’s and writes traverse the cache. As a read is observed it is retrieved from disk and subsequently stored for future use, meaning any subsequent read will be served directly from cache.

But read’s a boring, and everyone has a solution for read caching. ThinIO also treats this RAM cache as a storage area for write IO. Write IO is committed nearly instantly to the cache and no IO is sent down to the disk while free space is available in the cache.

“But what about if the machines run out of RAM?”

Well I’m glad you asked! The cache in ThinIO is hard set at a value you configure, so RAM will never be taken from the cache to service other processes. But in the situation where the cache has become 100% volatile write, ThinIO will then begin to spill over to the local disk allowing the Virtual Machine continue to operate.

There’s more, ThinIO actively manages cache contents to ensure it’s as relevant as possible. As the cache begins to fill, ThinIO’s Lazy Page Writer can identify and flush out blocks that have not been frequently used. This allows you to use a relatively small cache size while still deliver the big numbers we’ll discuss later.

Designed to be fool proof:

ThinIO’s GUI is fool proof, it’s intuitive and gives you a really quick view of ThinIO Realtime. ThinIO provides graphical representations of stats on reads, writes and cache usage, as well as an immediate view of the benefit ThinIO has created for the desktop.

The ThinIO console can also remotely connect to machines to ensure you don’t have to disturb the user while checking performance.

When the cache is enabled, ThinIO also has a realtime statistics window to help you identify disk patterns and cache performance

Boot and application launch time optimization:

ThinIO has some really clever technology built in to optimise the windows boot process and user experience.

During early testing, we observed just how inefficiently windows uses its disk resources during the boot process. Regularly the same files are requested over and over again on boot, if these blocks are non-contiguous, seek times are inherent. Busy servers were requesting up to 80,000 read IOPS during boot and process start.

ThinIO’s Read Ahead feature allows you to teach windows to be less of a storage monster. As the ThinIO cache is already aware of all blocks needed to boot, or even serve the users first launch of their applications, Read Ahead allows you to boot the machine, with a preloaded cache of required blocks, sorted contiguously.

When ThinIO starts up, it identifies the ‘Read Ahead’ configuration file and pauses windows while it reads the required blocks once, in a contiguous pattern around the disk. Once finished, windows continues to boot retrieving the majority of its block data directly from cache.

By doing this, ThinIO was delivering roughly 30% improved boot times while also reducing boot IOPS by over 80% in our testing. In the below graphs, we did a side by side comparison of the windows start-up process with and without ThinIO.

In the Gui below you will see a machine with the ThinIO cache enabled but no read ahead configured, we achieve a good 40% reduction of IOPS on boot and login, which is not bad on it’s own, but we knew we could make it better:

So after configuring a ‘Read Ahead’ configuration by booting a machine, logging in, opening the core set of applications and committing the file we see the following, large improvement of IOPS saving and cache hit rate on read:

So there you have it. By taking an additional 3-4 minutes with your golden image, you reduce nearly 30,000 IOPS to roughly 5,000 IOPS while also reducing boot times. Not only have you taken alot of pressure off your storage, if you launched your users applications core files as part of the read ahead configuration, your user’s login speeds will receive a really good boost while making their application launch times near instant.

Once the read ahead is complete, the driver will then start to use the data which is no longer needed for more chatty blocks of read or write, so configuring read ahead has zero impact on cache usage in the longer term.

Deploy, size, done.

Out of box, ThinIO takes less than 5 minutes to install and configure delivering you immediate benefits. No hoping, trusting or praying the hardware vendor’s figures are correct. No SAN or LUN type requirement, no hardware lead time, no hypervisor requirements and no change needed to your architecture. Whether you are doing on premises or even cloud SaaS / DaaS, ThinIO installs without any change.

Licensing:

ThinIO will ship with a 30 day grace period for you to test to your heart’s content without any commitment. If ThinIO is not for you, it’s just a matter of uninstalling it! Keeping in the spirit of the community, ThinIO will even have a free version available!

Ultimately, designing and deploying virtual desktops is difficult, we really wanted to write a product that both delivers and is simple and easy to deploy. We feel we’ve absolutely hit the mark on this and we look forward to opening the program to full deployment in the coming weeks.

Sounds great, how do I learn more?

Head on over to the ThinScale Technology web page and read more or register for the private beta.

Well we’ve been busy! Very, very busy. In the next week you will see the culmination of  two years work on a product we’re about to release called ThinIO.

Cast your mind back if you will to some ramblings and napkin math I devised some time ago in my series on IOPS negation strategies:

IOPS, Shared Storage and a Fresh Idea. (Part 1)
IOPS, Shared Storage and a Fresh Idea. (Part 2)
IOPS, Shared Storage and a Fresh Idea. (Part 3)

In these post’s a bunch of community guys ( Barry Schiffer, Iain Brighton, Ingmar Verheij, Kees Baggerman, Remko Weijnen and Simon Pettit) devised a cunning plan during E2EVC to see if we could counter the monotony of IOPS and their devastating impact on Virtual Desktop implementations. We threw together a loose test scenario where we could demonstrate how technology from Microsofts Windows Embedded Standard functionality EWF (Extended Write Filter) and Citrix’s XenServer intellicache with explosive performance and IO reduction statistics.

This blog series got way more attention than we possibly hoped and judging by citrix’s response by adding ram caching and disk overflow in Citrix provisioning services… we were definitely listened to. At the end of the series, I elluded to a technology that could be leveraged to achieve some of this, while right, it has taken along time to get right! With the help of our newest collaborator David Coombes, this technology is very much alive and ready for use.

Here’s the kicker:

Next week at Citrix Synergy, we’re dropping some big news for this market, we’re releasing a product that will deliver insanely fast IOPS to any storage utilising inexpensive RAM. With our product, no architecture change is required, no san volume dependencies, no expensive hardware upgrades and no hypervisor gotcha’s. ThinIO works with all major desktop virtualisation products like XenApp, XenDesktop, VDI in a Box, Microsoft Remote Desktop technologies and even VMware Horizon View!

ThinIO is just a simple installation and off you go. Not only will this product reduce, standardise and improve the speed of IOPS, it will also optimise and reduce boot storms dramatically.

Register for XenAppBlogs webinar here where we’ll discuss how ThinIO works for the first time or come visit us in Citrix Synergy (Booth 513) to celebrate the culmination of 2 years of work and learn how ThinIO is performant, reliable and an extremely cost effective method to deliver lightning fast experience to your users while protecting your disk storage from grinding to a halt.

Watch this space.

Register for Xenappblogs webinar with ThinScale Technology for the official launch of ThinIO

So after the first post, my mailbox blew up with queries on how to do this, how the ram cache weighs up vs pvs and how can you manipulate and “Write Out” to physical disk in a spill over, so before I go any further, lets have a quick look at the EWF.


Deeper into the EWF:


As previously mentioned, the EWF is contained in the operating system of all Windows Embedded Operating systems and also Windows Thin PC. The EWF is a mini filter driver that redirects all reads and writes to memory or local file system depending on where the file currently lives.

In the case of this experiment, we’re using the RAMREG method of EWF. Have a read of that link if you would like more information, but basically the EWF creates an overlay in RAM for the files to live and the configuration is stored in the registry.

Once the EWF is enabled and rebooted, on next login (from an adminstrative command prompt) you can run ewfmgr -all to view the current status of the EWF:





So what happens if I write a large file to the local disk?

Well this basically! Below I have an installer for SQL express which is roughly 250mb,  I copied this file to the desktop from a network share, as you can see below the file has been stuck into the RAM overlay.





And That’s pretty much it! Simple stuff for the moment. When I venture into the API of the EWF in a later post I’ll show you how to write this file out to storage if we are reaching capacity, allowing us to spill to disk and address the largest concern surrounding Citrix Provisioning Server and RAM caching presently.


Next up to the block. Machine Creation Services…


I won’t go into why I think this technology is cool, I covered that quite well in the previous post. But this was the first technology I planned to test with the EWF.

So I created a lab at home consisting of two XenServers and an NFS share with MCS enabled as  below:





Now before we get down to the nitty gritty, let’s remind ourselves of what our tests looked like before and after using the extended write filter in this configuration when booting using and shutting down a single VM:



As with last time, black Denotes Writes, Red Denotes Reads.

So looking at our little experiment earlier, we pretty much killed write IOPS on this volume when using the Extended write filter driver. But the read IOPS (in red above) were still quite high for a single desktop.

And this is where I had hoped MCS would step into the fray. Even on the first boot with MCS enabled the read negation was notable:

Test performed on a newly built host.

At no point do read’s hit as high as the 400 + peak we saw in the previous test. But we still see spikey read IOPS as the Intellicache begins to read and store the image.

So now that we know what a first boot will look like. I kicked the test off again from a pre cached device, this time the image should be cached on the local disk of the XenServer as I’ve booted the image a number of times.

Below are the results of the pre cached test:

Holy crap…

Well, I was incredibly impressed with this result… But did it scale?

So next up, I added additional desktops to the pool to allow for 10 concurrent desktop (restraints of my lab size).

Once all the desktops had been booted, I logged in 10 users and decided to send a mass shutdown command from Desktop studio, with fairly cool results:

And what about the other way around, what about a 10 vm cold boot?

Pretty much IO Free boots and shutdowns!

So lets do some napkin math for a second.

Taking what we’ve seen so far, lets get down to the nitty gritty and look at maximums / averages.

In the first test, minus EWF and MCS, from a single desktop, we saw:

• Maximum Write IOPS: 238
• Average Write IOPS: 24
• Maximum Read IOPS: 613
• Average Read IOPS: 83

And in the end result, with EWF MCS and even with the workload times 10, we saw the following:

• Maximum Write IOPS: 26 (2,380*)
• Average Write IOPS: 1.9 (240*)
• Maximum Read IOPS: 34 (6130*)
• Average Read IOPS: 1.3 (830*)

* Denotes original figures times 10

I was amazed by the results, of two readily available technologies coupled together to tackle a problem in VDI that we are all aware of and regularly struggle with.

What about other, similar technologies?

Now as VMware and Microsoft have their own technologies similar to MCS (CBRC and CSV cache, respectfully). I would be interested in seeing similar tests to see if their solutions can match the underestimated Intellicache. So if you have a lab with either of these technologies, get in touch and I’ll send you the method to test this.

What’s up next:

In the following two blog posts, I’ll cover off the remaining topics:

• VDI in a Box.
• EWF API for spill over
• Who has this technology at their disposal?
• Other ways to skin this cat.
• How to recreate this yourself for you own test.



Last up I wanted to address a few quick queries I received via email / Comments:


“Will this technology approach work with Provisioning Server and local disk caching, allowing you to leverage PVS but spill to a disk write cache?”

No, The Provisioning Server filter driver has a higher altitude than poor EWF, so PVS grabs and deals with the write before EWF can see it.

“Couldn’t we just use a RAM disk on the hypervisor?”

Yes, maybe and not yet…

Not yet, with Citrix MCS and Citrix VDI in a Box, Separating the write cache and identity disk from the LUN on which the image is hosted is a bit of a challenge.

Maybe If using Hyper-V v3 with the shared nothing migration, you now have migration options for live vm’s. This would allow you to move the WC / ID from one ram cache to another.

Yes, If using Citrix Provisioning server you could assign the WC to a local storage object on the host the VM lives. This would be tricky with VMware ESXi and XenServer but feel free to give it a try, Hyper-V on the other hand would be extremely easy as many ram disk’s are available online.

“Atlantis Ilio also has inline dedupe, offering more than just ram caching?”

True, and I never meant, even for a second to say this technology from Atlantis was anything but brilliant, but with RAM caching on a VM basis, wouldn’t VMware’s Transparent page sharing also deliver similar benefits? Without the associated cost?

Note: this is part 1 of a 4 post blog.

While attending E2EVC vienna recently, I found myself attending the Citrix Geekspeak session, although the opportunity to rub shoulders with fellow Citrix aficionado’s was a blast, I found myself utterly frustrated spending time talking about storage challenges in VDI.

The topic was interesting and informative, and while there were a few idea’s shared about solid state arrays, read IOPS negation with PVS or MCS there really wasn’t a vendor based, one size fits all solution to reduce both Read and Write IOPS without leveraging paid for (and quite expensive I may add) solutions like Atlantis ILIO, Fusion IO, etc. The conversation was tepid, deflating and we moved on fairly quickly to another topic.

So we got to talking…

In the company of great minds and my good friends Barry Schiffer, Iain Brighton, Ingmar Verheij, Kees Baggerman, Remko Weijnen and Simon Pettit. Over lunch we got to talking about this challenge and the pro’s and con’s to IO negation technologies.

Citrix Provisioning Server… 

We spoke about the pro’s and con’s delivered by Citrix provisioning server, namely rather than actually reducing read IOPS to shared storage, it merely puts the pressure instead on the network rather than a SAN or local resources in the hypervisor.

The RAM caching option for differencing is really clever with Citrix Provisioning server, but it’s rarely utilised due to the ability to fill the cache in RAM and the ultimate blue screen that will follow this event.

Citrix provisioning server does require quite a bit of pre work to get the PVS server configured, highly available then the education process of learning how to manage images with Provisioning server, it’s more of an enterprise tool.

Citrix Machine Creation Services…

We spoke about the pro’s and con’s of delivered by Citrix Machine Creation Services and Intellicache, this technology is much smarter around caching regular reads to a local device but it’s adoption is mainly SMB and the requirements for NFS and XenServer were a hard sell… Particularly with a certain hypervisors dominance to date.

Machine Creation Services is stupidly simple to deploy, install XenServer, stick the image on an NFS share (even windows servers can host these)… Bingo. Image changes are based on Snapshots so the educational process is a null point.

But again MCS is only negating read IO, and this assumes you have capable disk’s under the hypervizor to run multiple workspaces from a single disk or array. It’s also specific to XenDesktop, sadly. So no hosted shared desktop solution.

We mulled over SSD based storage quite alot with MCS and in agreement we discussed how MCS and Intellicache could be leveraged on an SSD, but the write cache or differencing disk activity would be so write intensive it would not be long before the SSD would start to wear out.

The mission Statement:

So all this got us to thinking, without paying for incredibly expensive shared storage, redirecting the issue onto another infrastructure service, or adding on to your current shared storage to squeeze out those precious few IOPS, what could be done with VDI and storage to negate this issue?

So we got back to talking about RAM:

RAM has been around for a long time, and it’s getting cheaper as time goes by. We pile the stuff into hypervisors for shared server workloads. Citrix provisioning server chews up the stuff for caching images in server side and the PVS client even offers a ram caching feature too, but at present it limit’s the customer to a Provisioning server and XenDesktop or XenApp.

But what if we could leverage a RAM caching mechanism? decoupled from a paid for technology, or provisioning server, that’s freely available and can be leveraged in any hypervisor or technology stack? Providing a catch all, free and easy caching mechanism?

and then it hit me…

Eureka!

This technology has been around for years, cleverly tucked away in Thin Client architecture and Microsoft have been gradually adding more and more functionality to this heavily underestimated driver…

Re Introducing ladies and gents, the Microsoft Extended write filter.

The Microsoft Extended Write Filter has been happily tucked away in the Microsoft embedded operating systems since XP and is fairly unfamiliar to most administrators. The Microsoft extended write filter saves all disk writes to memory (except for specific chunks of registry) resulting in the pc being clean each time it is restarted.

This technology has been mostly ignored unless you are particularly comfortable with Thin Client architectures. Interestingly during my initial research for this mini project, I found that many car pc enthusiasts or early SSD adopters have been hacking this component out of Windows Embedded and installing it into their mainstream windows operating systems to cut down on wear and tear to flash or Solid state disks.

Microsoft have been gradually building this technology with each release adding a powerful API to view cache hits, contents of the cache and even write the cache out to disk as required, in an image update scenario… or even better, when a RAM spill over is about to happen…

If this technology is tried and trusted in the car pc world, could it be translated to VDI?…

To the lab!

Excited with my research Thus far, I decided to take the plunge. I began by extracting the EWF drivers and registry keys necessary to run this driver from a copy of windows Embedded standard…

With a bit of trial, error, hair pulling and programming I managed to get this driver into a clean windows 7 x86 image, so from here I disabled the driver and started testing first without this driver.

(I’m not going to go into how to extract the driver yet, please check back later for a follow up post)

Test A: XenDesktop,  no RAM caching

To start my testing, I went with a XenDesktop image, hosted on XenServer without intellicache. I sealed the shared image on an NFS shared from a windows server. I performed a cold boot, opened a few applications then shutdown again.

I tracked the read and write IOPS Via Windows Perfmon on the shared NFS disk, and below were my findings:

(Black line denotes Write IOPS to shared storage)

Test B: XenDesktop, with the Microsoft Extended Write Filter.

The results below pretty much speak for themselves…

(again, black line denotes write IOPS)

I couldn’t believe how much of a difference this little driver had made to the write IOPS…

Just for ease of comparison, here’s a side by side look at what exactly is happening here:

So where do we go from here:

Well, no matter what you do, Microsoft will not support this option and they’d probably sue the pants off me if I wrapped this solution up for download…

But this exercise on thinking outside of the box raised some really interesting questions… Around other technologies and the offerings they already have in their stack.

In my next few posts on this subject I’ll cover the below topic’s at length and discuss my findings.

• I’ll be looking at how to leverage this with Citrix Intellicache for truly IO less desktops.
• I’ll be looking at how this technology could be adopted by Microsoft for their MED-V stack.
• I’ll be looking at one of my personal favourite technologies, VDI in a Box.
• I’ll be looking at how we could leverage this API ,for spill over to disk in a cache flood scenario or even a management appliance to control the spill overs to your storage.

And Finally, I’ll be looking at how Citrix or Microsoft could quite easily combine two of their current solutions to provide an incredible offering.

And that’s it for now, just a little teaser for the follow up blog posts which I’ll gradually release before Citrix Synergy.

I would like to thank Barry Schiffer, Ingmar Verheij, Kees Baggerman and Remko Weijnen. For their help and input on this mini project. It was greatly appreciated.