After creating an AppDisk, it got a little stuck. I tried deleting the task and AppDisk but the disk just seemed to be stuck in the “creating” phase indefinately.

To remove it, I had to do the following in Powershell from a delivery controller:

ASNP Citrix*
get-applibtask -active $true

Once i had identified the TaskID, i ran:

Stop-AppLibTask -TaskId 5be48afc-263b-454e-b3e9-5a2db6b966ff
remove-AppLibTask -TaskId 5be48afc-263b-454e-b3e9-5a2db6b966ff

Now just one last thing to do!

get-applibappdisk and find the entry in the “creating” state:Now finish with a:

remove-AppLibAppDisk -AppDiskUid d54681d3-6d0a-4259-b3ee-b90a899033bd

using the DiskUid that you saw in the previous get-applibappdisk command.

Thats it!

Update:

if you find that the machine you usually use to capture app disks is no longer available. use the following command to release it:

set-BrokerMachine -IsReserved $false -MachineName <MachineName>

PS: I could have piped a lot of that, sure! but my mac pipe symbol is currently out of action so i did it the hard way. if you are the type who reads others blogs and leaves comments to improve the scripting, I’m sure there are new people on the Microsoft forums you could be bawking at right now.

One area it’s always lacked is real time alerting. In order to really know what’s going on in your environment you need to be logged into director and watching. This is less than ideal and few monitoring vendors have endeavored to actually pull this data into their own solutions.

With the help of Rachel Berry, Prateek Kansal and Sridhar Mullapudi from Citrix. I set about diving into the logic and monitoring options within the FMA architecture. Citrix did a great job here and most if not all of it was readily available in PowerShell and oData. So, with the help of Citrix and a little bit of hard work, I’m very pleased to announce my latest free tool!

About the tool:

The Citrix Director Notification service sits on an edge server as a service (or local to the delivery controller) and periodically checks the health of:

• Citrix Licensing.
• Database Connections.
• Broker Service.
• Core Services.
• Hypervisor Connections.

And if any of these items fall out of bounds, an SMTP alert is sent to the mailbox of your choice for action. The tool will also send “All Clear” emails when these items are resolved, ensuring you are aware when the service has resumed a healthy state, neat huh?

An example of one of these alarm emails can be seen below:

Hypervisor Alerts, too!

Did you know that the Citrix XenDesktop and XenApp 7.x suite also keep track of Hypervisor alarms? It was news to me! Any-who, if you are lucky enough to be running XenServer or vSphere. The director Notification service can be configured to also catch these alarms, allowing you to also be alerted.

An example of a Hypervisor alarm is below:

Installation guide:

Below you’ll find the steps to get the director notification service installed:

 Download:

Director Notification Service 1.0.5

Setup:

• Download the Tool from the above URL.
• Install the service on a server running Windows Server 2008 R2 or above with .Net Framework 4.

Provisioning the Account:

The service account maintains a WinRM remote session to your delivery controllers and performs checks on a frequency you determine. Setting up this account is the meat of the install.

• Create an Active Directory Service account for monitoring director (this is needed later).
• Copy the XDServiceAccountProvision.ps1 file from the installation directory and run this script on all delivery controllers to automatically create the correct permissions for your service account.
• If you’re curious, read the script. A copy is below if you feel like doing it manually.

Using the configuration tool:

You’ll find the configuration utility on the start menu, under Direct Notify Configuration (this tool requires admin permissions).

Once open, configure the tool to suit your environment:

Ensure to use the test buttons.

Once happy all is working. Start or restart the “Director Notification Service”.

Advanced Logging:

A debugging option exists in the configuration tool, this will log to a trace.log in the installation directory of the machine. If you have issues, first enable this and review the log.

Provisioning Script contents:

write-warning "Before proceeding, Ensure you have: "
write-warning "1: Created a service account for monitoring XenDesktop."
write-warning "2: Ensure you have local administrative access to the local machine."
write-warning "3: Ensure this powershell instance is elevated."
write-warning "4: Ensure you have Administrative access to the XenDesktop site."
read-host "Press Any Key To Continue"

asnp citrix* -ea 0

$domain=Read-Host -Prompt "enter domain name the monitoring user is a member of: e.g. lab"
$username=Read-Host -Prompt "enter monitoring user name: e.g. john.doe"
$group="Remote Management Users"

if(!(Get-AdminRole "Direct Notify Role")){
#create new administrative role
new-adminrole -Name "Direct Notify Role" -Description "used for Direct Notify Service"
Add-AdminPermission -Role "Direct Notify Role" -Permission Configuration_Read
Add-AdminPermission -Role "Direct Notify Role" -Permission Configuration_Write # needed for some unknown reason, thanks citrix ¯\_(ツ)_/¯
Add-AdminPermission -Role "Direct Notify Role" -Permission EnvTest
Add-AdminPermission -Role "Direct Notify Role" -Permission Global_Read
Add-AdminPermission -Role "Direct Notify Role" -Permission Hosts_Read
Add-AdminPermission -Role "Direct Notify Role" -Permission Licensing_Read

#create administrator
New-AdminAdministrator -Name $domain\$username
add-adminright -Administrator $domain\$username -Role "Direct Notify Role" -Scope all
}
#configure remote management if not configured
winrm quickconfig -quiet

#add the monitoring user to the remote management group
([ADSI]"WinNT://$env:computername/$Group,group").psbase.Invoke("Add",([ADSI]"WinNT://$domain/$username").path)

 Fine Print:

Just a few things to be aware of:

Application Dependencies:

• The Notification service requires .Net 4.0
• PowerShell Remoting must be enabled on the Broker Servers
• A service account should be created to monitor the brokers (included in the script).

Tested on:

• Citrix XenDesktop 7.6.
• (Previous versions should work fine too.)

Support:

Drop me an email on andrew@andrewmorgan.ie if you have any trouble!

Enjoy and i hope this is very useful to you.

Future plans:

I’ll be diving into the oData values in the Citrix monitoring database in the next itteration. This work has already begun and will be updated soon.

ThinKiosk 4.5 is a big update, so without further ado, lets get right to it:

ThinKiosk Broker Service:

• HA features are now available in the ThinKiosk Broker.
• The Broker service can now utilise Microsoft SQL for the database and an easy migration utility can be utilised to do so.
• Brokers can now be load balanced via Citrix Netscaler or Microsoft DNS round robin.
• The ThinKiosk Broker can now deliver software updates directly to clients.
• The ThinKiosk Broker can now authenticate against active directory.

ThinKiosk Client:

• The ThinKiosk receiver functionality has been moved to the applications tab in our new ThinScale Connector functionality.
• The Client can now communicate directly with Microsoft RDS Broker services. Allowing customers using Microsoft RDS or VDI to use ThinKiosk to connect, enumerate and launch resources within ThinKiosk, without RDP files.
• The Client now supports password changing and legal notices for the ThinScale Connector.
• The Client now supports “auto launch” logic to specify which desktops to auto launch on logon.
• The Client now starts up at least 40% quicker than previous versions.
• The Client’s communication logic has been redesigned to allow management even when nobody is logged in.
• The Client now supports central software updates from the Broker, allowing push software updates.
• The Client now has an authentication API for use with Imprivata tap and go cards or similar technology.
• The Client is now smart enough to detect DNS round robin when connecting to a Broker and will use the list retrieved from DNS as a broker list to try when starting up.
• The Clients will delete stale or old user profiles periodically to keep machines clean.
• Many, many improved administrative features allowing ease of access to the system for administrators.

We’re extremely proud of this update and we look forward to hearing from you!

I had neglected to update this tool for a while, until I actually needed it and the remote screen saver annoyed the hell out of me. necessity is the mother of product maintenance it seems!

Anyway, I digress, check the original blog post here for the downloads and configuration options.

In other news, if you’re familiar with ThreadLocker, watch this space, it’s about to get a serious overhaul!

PS: stop asking me for a mac client, it’s not possible as there is no ICA SDK / API for mac.

• ThinIO facts and figures, Part 1: VDI and Ram caching.
• ThinIO facts and figures, Part 2: The Bootstorm chestnut.
• ThinIO facts and figures, Part 3: RDS and Ram caching.

In the final blog post in this series, we’re going to discuss storage design and a frequent problem face when sizing storage. Lets get right into it:

“Designing for average, is designing for failure”

Peak IOPS:1015, Average IOPS: 78

A frequent mistake I see customers and consultants alike make is taking an average of a sizing requirement and using that as a baseline for sizing environments.

Looking at the figures produced from our internal load tests, we saw just an average of 78 IOPS required on write from a vanilla server without ThinIO to provide storage from this XenApp server.

Now frequently, people will take a figure like that, throw in 25% for growth and bob’s your uncle, ‘order the hardware Mr SAN man’. When I have questioned them about that assumption, they’ll often respond “oh it will be a bit slow if theres contention but it’ll even itself out”.

Right? Wrong.

Things don’t go slow when they are over subscribed, they stop.

Don’t take my word for it! Lets do some simple theoretical math:

If you take a storage device and allocate 100 IOPS to this machine, what’s going to happen when a peak like 1000 IOPS is requested? A lot of queuing. 

In theory, keeping to the 100 IOPS figure, that 1 second burst IO is now taking over 10 seconds to satisfy (1000 / 10).

But it gets worse, all subsequent IO that is requested after that spike occurred is going to also be haulted waiting for this task to occur.

Assuming you’re now mid spike and 10 seconds later the request is finished… taking your average figure, you now have 10 seconds worth of 100 IO’s per second potentially queued up behind…

Low and behold another login occurs and? STOP. Storage timeouts, twirly whirlies, application crashes, hour glasses and the good old “logins are slow”.

Oh ok, So how do I size to accommodate this? 

Well you’re between a rock and a hard place aren’t you. You can’t tell users when to login, the price tag of a SAN sized for peak activity + 20% is going to cost you more than your entire desktop estate. And as you can see, it’s never safe to assume it will run slow. 

Buying into shared storage is a tricky business 

Storage is expensive. Very expensive. It always annoys me when you hear vendors in this space refering to themselves as ‘reassuringly expensive’. To me this directly translates to ‘We can charge what we want, so we will and you can be reassured we feel the price tag is worth it.’

Storage was never written with desktop workloads in mind, it was written for ‘steady state’ server workloads and was in the progress of going the way of the ‘dodo’ (extinct) up until that first release of vMotion requiring shared storage, which some say was the saving of the market.

Many vendors are going with software or hardware intelligent tiering. This is a great feature, but the real question to ask is how frequently data is moved from the hot tier to lower tier? Press your vendor on this as they more than likely wont know! Microsoft storage spaces is a prime example of this with a really poor optimisation process of just once a day!

Then ask yourself what happens when a base image update occurs and changes the disk layout of the base golden image? Further, stateless technologies from the bigger vendors delete the differencing disk on restart, can you be sure the new disk is going to end up in the smaller, faster SSD or RAM tier? Or is data up there already in contention?

RAM is far less tricky

RAM is commodity, available in abundance and throughout every virtual desktop project I’ve architected and deployed, you run out of CPU resources in a ‘fully loaded’ host way before you will run out of RAM. RAM has no running maintenance cost, Ram is an upfront CAPEX cost and requires little to no maintenance.

The beauty of what ThinIO does with the little resources you assign it, is turn that desktop workload into a healthier and happier server workload of minimal burst IO and a low steady state IO requirement.

Note the peak of just 40.5 IOPS and average IOPS of less than 2.

with as little as just 200mb cache for each of the 10 users logging in, within an aggressive 3 minute window, we reduced the peak from 1000 to 40. That’s a 96% reduction in burst IO.

With ThinIO, you:

• reduce your exposure to massive IO spikes.
• Improve user logon times.
• significantly reduce your daily IOPS run rate.
• Increase user productivity by spending less time waiting for the storage.
• Commit to nothing up front, test it and see how well it works. If you are happy, then buy in.

Lots of Intelligence baked in:

ThinIO is acutely aware of key operating system events that will cause these kind of spikes and react accordingly to reduce the spikes in IOPS created. ThinIO constantly watches the behavior and IO pattern of the storage and tunes itself accordingly.

Unlike other technologies, ThinIO is a true caching and performance solution. We do not move useful data in and out of the cache on demand when cache availability is contencious. We track patterns and frequency of block access to respond accordingly, delivering all the benefits we have mentioned, even with the tiniest cache, without EVER reducing the capability of the storage when overwhelmed.

And on the opposite side of the scale, when underworked, we leverage our cache to deliver deeper read savings as above.

ThinIO also has a powerful API and PowerShell interface to allow you to report and interact with the cache on demand.

Wrap up:

And with the end of the series looming, allow me to finish on some easy points:

 ThinIO Allows you to:

• size your SAN outside of the Lamborghini category & price tag for your desktop estate.
• rapidly achieve far deeper density on your current hardware when you are feeling the Pinch.
• guarantee a level of performance by assigning cache per VM, disallowing other users to steal or hamper caching resources.
• Improve user experience and login times immediately.
• Reduce the impact of boot storms and similar IO storm scenarios.

No other vendor can offer as quick a turn around time with their product. ThinIO installs in seconds and offers a huge range of compatibility.

One more thing:

In case you missed ThinIO’s launch day at E2EVC Barcelona, ThinIO is now in GA, available from our website and production ready! More marketing to follow! But grab your copy now and get playing!

To recap, below you will find the previous articles:

• ThinIO facts and figures, Part 1: VDI and Ram caching.
• ThinIO facts and figures, Part 2: The Bootstorm chestnut.

Off topic note:

two years ago at an E2EVC event, the concept behind ThinIO was born with just a mad scientist idea amongst peers.

If you are lucky enough to be attending E2EVC this weekend, David and I will be there presenting ThinIO and maybe, just maybe there will be an announcement. Our session is on Saturday at 15:30 so pop by, you won’t be disappointed.

Back on topic:

So here’s a really interesting blog post. Remote Desktop Services (XenApp / XenDesktop hosted shared) or whatever you like to call it. RDS really presents a fun caching platform for us, as it allows us to deal with a much higher IO volume and achieve deeper savings.

We’ve really tested the heck out of this platform for how we perform on Microsoft RDS, Horizon View RDS integration and Citrix XenSplitPersonality with Machine Creation Services.

The figures we are sharing today are based on the following configuration and load test:

• Citrix XenDesktop 7.6
• Windows Server 2012 r2
• Citrix User Profile Manager.
• 16gb of Ram.
• 4 vCpu.
• LoginVSI 4.1 medium workload 1 hour test.
• 10 users.
• VMFS 5 volume.

Fun figures!

Diving straight in, lets start by looking at the volume of savings across three cache types.

Reviewing the details for a moment:

Running repetitive tests of at least 3 per cache type, we found even at the lowest entry point we would support (50mb per user) we saw phenomenal savings of over 70% on write IO.

No pressure no diamonds!

To put that into perspective, at a 512 MB cache for 10 users, our cache reached maximum capacity at the second user login. With 8 users still left to login, cache full and still an hours worth of load testing left, our ThinIO technology was under serious pressure.

This is key to why ThinIO is such a great solution. We won’t just perform great until we fill our cache, we don’t require architecture changes or care about your storage type, we have no lead times or install days, we will carry on to work with what is available to use, to take a large ammount of pressure off storage IOPS and data throughput.

With the figures above, you can see just how well the intelligence behind our cache can scale even when it faces such a steep workload.

Below you will find a breakdown of each test:

512 MB cache:

Breaking down into the figures, on the 512mb cache test, it’s clear to see just how well ThinIO deals with the tiniest of caches:

When we side by side this with our baseline averages, you can see we take a huge chunk out of that Spiky login pattern and continue to  reduce the steady state IO as the test continues:

So lets move up and see how we get on!

1024 mb cache:

Doubling up our cache size we see a great increase in both read and write savings as you’d expect.

With 100mb of cache per user, and the average user profile in the test 3 times that size. We are still under pressure. As we will natively favour optimisations to write IO over read, you’ll see the bulk of improvements happen in write when we’re under pressure as illustrated in this test:

With more cache available during the peak IO point, we make further savings on write:

2048 mb cache:

and at our recommended value of 200mb per user in Remote Desktop Services, the results are phenomenal! With this size, even still below the 300mb mark per user profile, the read IO gets a really good boost and the write IO saving well over the 95% mark!

And the side by side comparison is every bit as good as the savings illustrated above, reducing that peak bursty IO to just 41 IOPS:

But there’s more! 

As i pointed out in the previous blog, IOPS are just one side of the story. A reduction of data throughput to the disk is also a big benefit when it comes to storage optimisation, and as you can see we make a big difference:

Wrap up:

So there you have it, with ThinIO, a simple, in VM solution, you can you seriously reduce your IO footprint, boost user performance and achieve greater storage density per virtual machine or on Remote Desktop Services technology.

In the mean time:

If you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

• ThinIO facts and figures, Part 1: VDI and Ram caching.

Getting right to it:

In this industry when somebody says ‘boot storms!’ – most of us will respond with:

Boot storms are a well documented, boring problem and have many solutions available from vendors and hypervisors alike. Most solutions today rely on a ‘shared memory’ storage area to cache ‘on boot’, in theory caching only one startup or one pattern in order to then serve it back to the proceeding desktops to boot.

But why are boot storms an issue? While working on ThinIO we had the unique ability to really dive into the Windows boot process and analyse why boot storms cause the damage they do and in this post we thought we’d share our findings to better document the issue.

Boot data:

Taking a typical windows 7 boot, to the login screen and idling until all services have started, the data traversing from disk to VM is relatively small. in our testing we found an average of Just 500-600 mb of data is read during this process, and write data barely registers at between 20 and 30mb.

But hey, what gives? Taking such low data throughput, why is boot such a contenscious issue? Have I been misled with marketing and vendor nonsense?

The IO chestnut:

Sadly no, it’s the way windows requests this data, but don’t take my word for it…. Behold, the incredible mess that is the Windows boot process!

Yep, that’s right, in the time Windows requested roughly 600mb of data, it sent down an astounding 70 thousand IO’s in the space of 2-3 minutes!

Math time:

Now if you were to take these figures as they stand, you would take 70,000 IO’s divide this into 560mb and you’d probably end up with an average of about 8k of data requested per IO… You’d be wrong.

As my good buddy Conor Scolard would say, ‘when you Assume, you make an ass out of you and me’.

To better understand the bounderies of Windows, Windows requests IO’s between the minimum of 512 bytes all the way up the spectrum later in the boot process to 128k and above. But it requests these blocks sparcely, on demand, and not just once per sector, the same blocks are frequently accessed.

The net result of this causes absolute havok on the storage:

The crux of the issue is, for each one of these IO’s, the storage provider needs to compute the block data requested, seek the data out, then return it.

But 70,000 of these IO operations for a meagre 600mb of data is madness and you can now see exactly why boot storms were labelled as such for those early adopters who had their hands burned by this fact finding mission.

I’ll mitigate this issue by just booting my VM’s at night!

I’m sure you will! I would also love to see your face if a number of users happen to restart their desktops during the day, cascading 70,000 IO’s per desktop to the storage in a 2 minute window, per desktop!

Bootstorming IS an issue.

Now, knowing all this, it makes sense as to why storage and hypervisors alike are using a cache of ram.

But how does ThinIO fit in here? With Read Ahead of course!

Knowing the Windows boot process as intimate as only a technology like ThinIO can, there are many, many optimisations we can make to this process.

We can both speed the boot process up and also massively reduce the storage requirement while in VM, without any fancy caching mechanism!

With ThinIO’s read ahead technology, we can deliver just shy of an 80% boot IO reduction with nothing other than having our technology in the virtual machine:

Taking a ThinIO averaged test and overlaying it to a baseline averaged test, it’s clear just how much impact this technology can have:

Wrap up:

So there you have it, with ThinIO, a simple, in VM solution, not only can you seriously reduce your IO footprint, boost user performance and achieve greater storage density per virtual machine, you also can also massively negate the impact a booting VM has on your storage.

If you would like a chance to test ThinIO pre-release, find access to the public beta below. Thank you for your time and happy testing!

Warm up your labs or fire up your golden images ladies and gents, we’re delighted to announce ThinIO’s brief public beta will begin today!

This project has taught us some really interesting things about Windows IO, how Windows behaves and how the hypervisor and storage can behave. This project really felt like a David vs. Goliath task as we (members of our community with a desire to simplify this issue) attempted to tackle one of the largest issues in our industry, storage bottlenecks and Windows desktops.

What’s really unique about our approach is there are no hardware lead times, no architecture changes needed and no external dependencies. ThinIO can be installed in seconds and the benefits are seen immediately.

We’ve spent countless hours testing, tuning, retesting and even more tuning. We’re extremely happy with the results. This public beta will serve as an opportunity for you to really kick the tyres and believe the hype in what we’ve built while we’re putting together the final touches to release the product in the coming weeks.

During this time, we found achieving positive and consistent IO negation boils down to a number of items:

• cutting down on the volume of IOPS sent to the storage.
• Reducing the data transferred (MB/sec) to and from the storage.
• Intelligently cutting down on peak IO, such as boot and user logon.

In the coming days we’re going drill down into these categories in more depth. But as a quick overview, here’s a baseline (top) and ThinIO (bottom) session comparison of a Windows 8.1 desktop login, 1 hour Login VSI medium workload and log off with just 350 mb of cache for ThinIO:

Keep an eye out for the coming blog posts, but in the mean time, the ThinIO beta is available to download here now! Go forth and have fun.

Until next time,

A

I wrote a tool quite some time ago called watcher2 while troubleshooting a similar issue. I finally took the time to refactor that tool for use with XenApp 6.5 , XenDesktop and VMware View and they are finally available to download! Both watcher utilities also include a latency counter which was a request that came in over and over.

HDX and PCOIP watcher by default dock to the top of the screen and can be moved left or right as below:

They can now also be completely un docked:

How do they work?

The tool finds your username in the performance monitor counters for session bandwidth, once it finds this entry it reads your performance monitor data once every second and reports on it.

In the case of PCOIP watcher, it reads the PCOIP counters from performance monitor.

what do the values mean?

All values are in either Kilobits per second or Megabits per second.

In = Traffic from the client to the virtual, this may spike during large copy / paste jobs,web cams or copying data from a usb key to the session:
Out = Traffic from the virtual desktop to the client, mainly audio or video traffic causes this to spike.
Latency = The delay between your client and the virtual desktop.

Can I Configure it?

Two thresholds are available, a yellow warning and a red warning, currently . These default values can be written to:

“HKEY_local_machine\software\ThinScaleTech\HDXmonitor”
“HKEY_current_user\software\ThinScaleTech\HDXmonitor”

Dword: YellowWarning = 300 (decimal)

Dword: RedWarning = 600 (decimal)

Do they have any dependencies?

.net framework 3.5

if you are running XenApp 6.5 or XenDesktop 5.6, ensure you have the latest hot-fixes installed or the counters may be incorrect.

How do I launch it?

Allow the user to run it manually, or place the executable in their start-up folder or login script.

Where Can I download it?

Here:

What’s coming next:

• Native Microsoft RDP Counters.
• Realtime graphs and recording.
• source code is available on request.

Iconizer:

Iconizer Converts png files (with transparency supported) to an Ico file format , then in turn converts it To a Base 64 String.

You can send the data to the clipboard or import directly into XenDesktop if you have the powershell tools available.

It’s very simple, I wont bore you with the details, just convert and import. then map with powershell:

Reverse Iconizer:

I’m sure you can guess, takes the massive string of information stored in base64 and gives you a visual representation.

An example command line of how to do this is below:

Hey wait?

Why didn’t you integrate both of these?

Well it seems .net and Powershell have a limit on the data (string length) it can pull out of the pipeline. The default Citrix icon is close to 20,000 characters and results in you being unable to pull this data from powershell directly to .net. WIth great help from http://www.jonathanmedd.net/  we found that, yep the console does seem to have a roughly 8k char limit… Sure I could parse it to a file or the clipboard, but that was messy and frankly, I really couldn’t be arsed.

If you are up to the challenge I’ve got the source code for forward and reverse of the icon data below. I’ve also got a half assed attempt at creating a list… So fill your boots and take up the challenge if you wish.

Download:

As with most of my utilities the download links and source code are below, and a few icons to get you going:

Download utilities.

Download Source Code.