• ThinIO facts and figures, Part 1: VDI and Ram caching.
• ThinIO facts and figures, Part 2: The Bootstorm chestnut.
• ThinIO facts and figures, Part 3: RDS and Ram caching.

In the final blog post in this series, we’re going to discuss storage design and a frequent problem face when sizing storage. Lets get right into it:

“Designing for average, is designing for failure”

Peak IOPS:1015, Average IOPS: 78

A frequent mistake I see customers and consultants alike make is taking an average of a sizing requirement and using that as a baseline for sizing environments.

Looking at the figures produced from our internal load tests, we saw just an average of 78 IOPS required on write from a vanilla server without ThinIO to provide storage from this XenApp server.

Now frequently, people will take a figure like that, throw in 25% for growth and bob’s your uncle, ‘order the hardware Mr SAN man’. When I have questioned them about that assumption, they’ll often respond “oh it will be a bit slow if theres contention but it’ll even itself out”.

Right? Wrong.

Things don’t go slow when they are over subscribed, they stop.

Don’t take my word for it! Lets do some simple theoretical math:

If you take a storage device and allocate 100 IOPS to this machine, what’s going to happen when a peak like 1000 IOPS is requested? A lot of queuing. 

In theory, keeping to the 100 IOPS figure, that 1 second burst IO is now taking over 10 seconds to satisfy (1000 / 10).

But it gets worse, all subsequent IO that is requested after that spike occurred is going to also be haulted waiting for this task to occur.

Assuming you’re now mid spike and 10 seconds later the request is finished… taking your average figure, you now have 10 seconds worth of 100 IO’s per second potentially queued up behind…

Low and behold another login occurs and? STOP. Storage timeouts, twirly whirlies, application crashes, hour glasses and the good old “logins are slow”.

Oh ok, So how do I size to accommodate this? 

Well you’re between a rock and a hard place aren’t you. You can’t tell users when to login, the price tag of a SAN sized for peak activity + 20% is going to cost you more than your entire desktop estate. And as you can see, it’s never safe to assume it will run slow. 

Buying into shared storage is a tricky business 

Storage is expensive. Very expensive. It always annoys me when you hear vendors in this space refering to themselves as ‘reassuringly expensive’. To me this directly translates to ‘We can charge what we want, so we will and you can be reassured we feel the price tag is worth it.’

Storage was never written with desktop workloads in mind, it was written for ‘steady state’ server workloads and was in the progress of going the way of the ‘dodo’ (extinct) up until that first release of vMotion requiring shared storage, which some say was the saving of the market.

Many vendors are going with software or hardware intelligent tiering. This is a great feature, but the real question to ask is how frequently data is moved from the hot tier to lower tier? Press your vendor on this as they more than likely wont know! Microsoft storage spaces is a prime example of this with a really poor optimisation process of just once a day!

Then ask yourself what happens when a base image update occurs and changes the disk layout of the base golden image? Further, stateless technologies from the bigger vendors delete the differencing disk on restart, can you be sure the new disk is going to end up in the smaller, faster SSD or RAM tier? Or is data up there already in contention?

RAM is far less tricky

RAM is commodity, available in abundance and throughout every virtual desktop project I’ve architected and deployed, you run out of CPU resources in a ‘fully loaded’ host way before you will run out of RAM. RAM has no running maintenance cost, Ram is an upfront CAPEX cost and requires little to no maintenance.

The beauty of what ThinIO does with the little resources you assign it, is turn that desktop workload into a healthier and happier server workload of minimal burst IO and a low steady state IO requirement.

Note the peak of just 40.5 IOPS and average IOPS of less than 2.

with as little as just 200mb cache for each of the 10 users logging in, within an aggressive 3 minute window, we reduced the peak from 1000 to 40. That’s a 96% reduction in burst IO.

With ThinIO, you:

• reduce your exposure to massive IO spikes.
• Improve user logon times.
• significantly reduce your daily IOPS run rate.
• Increase user productivity by spending less time waiting for the storage.
• Commit to nothing up front, test it and see how well it works. If you are happy, then buy in.

Lots of Intelligence baked in:

ThinIO is acutely aware of key operating system events that will cause these kind of spikes and react accordingly to reduce the spikes in IOPS created. ThinIO constantly watches the behavior and IO pattern of the storage and tunes itself accordingly.

Unlike other technologies, ThinIO is a true caching and performance solution. We do not move useful data in and out of the cache on demand when cache availability is contencious. We track patterns and frequency of block access to respond accordingly, delivering all the benefits we have mentioned, even with the tiniest cache, without EVER reducing the capability of the storage when overwhelmed.

And on the opposite side of the scale, when underworked, we leverage our cache to deliver deeper read savings as above.

ThinIO also has a powerful API and PowerShell interface to allow you to report and interact with the cache on demand.

Wrap up:

And with the end of the series looming, allow me to finish on some easy points:

 ThinIO Allows you to:

• size your SAN outside of the Lamborghini category & price tag for your desktop estate.
• rapidly achieve far deeper density on your current hardware when you are feeling the Pinch.
• guarantee a level of performance by assigning cache per VM, disallowing other users to steal or hamper caching resources.
• Improve user experience and login times immediately.
• Reduce the impact of boot storms and similar IO storm scenarios.

No other vendor can offer as quick a turn around time with their product. ThinIO installs in seconds and offers a huge range of compatibility.

One more thing:

In case you missed ThinIO’s launch day at E2EVC Barcelona, ThinIO is now in GA, available from our website and production ready! More marketing to follow! But grab your copy now and get playing!

• ThinIO facts and figures, Part 1: VDI and Ram caching.

Getting right to it:

In this industry when somebody says ‘boot storms!’ – most of us will respond with:

Boot storms are a well documented, boring problem and have many solutions available from vendors and hypervisors alike. Most solutions today rely on a ‘shared memory’ storage area to cache ‘on boot’, in theory caching only one startup or one pattern in order to then serve it back to the proceeding desktops to boot.

But why are boot storms an issue? While working on ThinIO we had the unique ability to really dive into the Windows boot process and analyse why boot storms cause the damage they do and in this post we thought we’d share our findings to better document the issue.

Boot data:

Taking a typical windows 7 boot, to the login screen and idling until all services have started, the data traversing from disk to VM is relatively small. in our testing we found an average of Just 500-600 mb of data is read during this process, and write data barely registers at between 20 and 30mb.

But hey, what gives? Taking such low data throughput, why is boot such a contenscious issue? Have I been misled with marketing and vendor nonsense?

The IO chestnut:

Sadly no, it’s the way windows requests this data, but don’t take my word for it…. Behold, the incredible mess that is the Windows boot process!

Yep, that’s right, in the time Windows requested roughly 600mb of data, it sent down an astounding 70 thousand IO’s in the space of 2-3 minutes!

Math time:

Now if you were to take these figures as they stand, you would take 70,000 IO’s divide this into 560mb and you’d probably end up with an average of about 8k of data requested per IO… You’d be wrong.

As my good buddy Conor Scolard would say, ‘when you Assume, you make an ass out of you and me’.

To better understand the bounderies of Windows, Windows requests IO’s between the minimum of 512 bytes all the way up the spectrum later in the boot process to 128k and above. But it requests these blocks sparcely, on demand, and not just once per sector, the same blocks are frequently accessed.

The net result of this causes absolute havok on the storage:

The crux of the issue is, for each one of these IO’s, the storage provider needs to compute the block data requested, seek the data out, then return it.

But 70,000 of these IO operations for a meagre 600mb of data is madness and you can now see exactly why boot storms were labelled as such for those early adopters who had their hands burned by this fact finding mission.

I’ll mitigate this issue by just booting my VM’s at night!

I’m sure you will! I would also love to see your face if a number of users happen to restart their desktops during the day, cascading 70,000 IO’s per desktop to the storage in a 2 minute window, per desktop!

Bootstorming IS an issue.

Now, knowing all this, it makes sense as to why storage and hypervisors alike are using a cache of ram.

But how does ThinIO fit in here? With Read Ahead of course!

Knowing the Windows boot process as intimate as only a technology like ThinIO can, there are many, many optimisations we can make to this process.

We can both speed the boot process up and also massively reduce the storage requirement while in VM, without any fancy caching mechanism!

With ThinIO’s read ahead technology, we can deliver just shy of an 80% boot IO reduction with nothing other than having our technology in the virtual machine:

Taking a ThinIO averaged test and overlaying it to a baseline averaged test, it’s clear just how much impact this technology can have:

Wrap up:

So there you have it, with ThinIO, a simple, in VM solution, not only can you seriously reduce your IO footprint, boost user performance and achieve greater storage density per virtual machine, you also can also massively negate the impact a booting VM has on your storage.

If you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

Test, test, review and tune. Rinse and repeat!

We’ve spent months load testing, tuning, fixing and retesting ThinIO. And for the first time we’re going to start talking about the dramatic results ThinIO can have on storage scalability and user perceived performance.

During our extensive testing cycles, we’ve covered:

• Horizon View
• Citrix XenDesktop
• Microsoft RDS

We’ve been seeing very similar, if not identical results when testing against pools on the following storage types too:

• XenServer SR
• VMFS
• NFS
• Microsoft Clustered Shared Volumes

For reference the statistics we’re sharing today are based on VDI via VMware Horizon View 6, these figures are averages across at least three independent tests. All details of the tests that exported these results are covered below.

Testing details:

The VM’s we tested in this particular workload are as follows:

• Testing Method: Login VSI 4.1 Medium Workload
• Operating System: Windows 7 x64 SP1
• System ram: 3gb
• vCPU: 2
• ThinIO cache: 350mb
• Technology:  VMware Horizon View 6
• Test runtime: 1 hour*
• Statistic sample period: 5 seconds.

With that out of the way, lets jump right in!

Storage IO:

The number of IO’s per second is crucially important when dealing with storage; many, many small IO’s sent to sparse locations on disk are a killer to storage technologies, only made worse by certain file systems.

As a storage acceleration and negation technology, were extremely happy with the IO’s reduction we see on the storage:

Even with just 350 MB of ram as a cache, we achieve phenomenal IO reduction.

Storage MB / Sec:

But IO’s are just one part of the puzzle, what about the size of the data requests being sent to the storage?

A true solution to take the pressure off the SAN, improve user performance, and increase storage density needs to tackle both the IO and the throughput problem.

As you can see above, with just 350mb we’re very good at it!

Side by Side Comparison:

So rounded figures are fine so long as the data is trustworthy, but here’s a real preview laid bare for your analysis.

Here’s a direct comparison on NFS and VMFS of taking a standard load test IO pattern and comparing it to an identical test with ThinIO installed in the VM:

VMFS:

NFS:

As you can imagine, we’re extremely proud of what we can achieve with as little as 350mb per desktop.

The beauty of our approach is simplicity, your users can see this benefit not in a matter of weeks, days or even hours. ThinIO can be up and running in minutes, delivering reduced login times, storage acceleration and providing a far deeper density on your current storage.

Wrap up:

So there you have it, we’ll be adding additional blog posts in the coming days looking at Remote Desktop Services / XenApp, intelligent cache management built in and our Read Ahead technology, so check back. In the mean time, if you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

* Eight-hour figures and complete statistics are also available, we have nothing to hide and we’d encourage you to get in touch with the ThinScale team and we’ll share them with you.

I wrote a tool quite some time ago called watcher2 while troubleshooting a similar issue. I finally took the time to refactor that tool for use with XenApp 6.5 , XenDesktop and VMware View and they are finally available to download! Both watcher utilities also include a latency counter which was a request that came in over and over.

HDX and PCOIP watcher by default dock to the top of the screen and can be moved left or right as below:

They can now also be completely un docked:

How do they work?

The tool finds your username in the performance monitor counters for session bandwidth, once it finds this entry it reads your performance monitor data once every second and reports on it.

In the case of PCOIP watcher, it reads the PCOIP counters from performance monitor.

what do the values mean?

All values are in either Kilobits per second or Megabits per second.

In = Traffic from the client to the virtual, this may spike during large copy / paste jobs,web cams or copying data from a usb key to the session:
Out = Traffic from the virtual desktop to the client, mainly audio or video traffic causes this to spike.
Latency = The delay between your client and the virtual desktop.

Can I Configure it?

Two thresholds are available, a yellow warning and a red warning, currently . These default values can be written to:

“HKEY_local_machine\software\ThinScaleTech\HDXmonitor”
“HKEY_current_user\software\ThinScaleTech\HDXmonitor”

Dword: YellowWarning = 300 (decimal)

Dword: RedWarning = 600 (decimal)

Do they have any dependencies?

.net framework 3.5

if you are running XenApp 6.5 or XenDesktop 5.6, ensure you have the latest hot-fixes installed or the counters may be incorrect.

How do I launch it?

Allow the user to run it manually, or place the executable in their start-up folder or login script.

Where Can I download it?

Here:

What’s coming next:

• Native Microsoft RDP Counters.
• Realtime graphs and recording.
• source code is available on request.