Everyone having a Good Citrix Synergy week? Some great new products announced! Ready for more announcements?

Great!

After 5 months of coffee, tears of frustration and hair pulling we’re absolutely delighted, thrilled and relieved to announce ThinKiosk 4.0 is nearly ready. Complete with my new partner in crime Remko Weijnen (I’ve been saying ‘we’ for ages, now you know who… awesome eh?) we’ve worked some long nights to get this version out the door.

With that out of the way, we’re proud to announce some of the new features coming in 4.0. Bear in mind this is just a preview, the final features and details of the product are still being hammered out, but below is a taster of some of the functionality you can expect to see shortly.

Back to the drawing board:

ThinKiosk 4.0 is a complete rewrite and refactor of ThinKiosk. It’s built on the 4.0 .Net framework which has brought a lot of simplicity and new features to our tool-set. ThinKiosk 4.0 was built with three main aims:

• Enterprise Ready.
• Fool Proof.
• Secure by Design.

With ThinKiosk 4.0, your setup time will go from days to minutes. Out of the box, ThinKiosk is ready for the following technologies without any local machine tuning:

• Citrix XenDesktop / XenApp.
• Citrix VDI in a Box.
• VMware View.
• Microsoft Remote Desktop Services.

For the exact details of each of these optimizations, follow the subsequent blog posts / documentation.

New Look and Feel:

Without further ado, lets start with the new look and feel:

ThinKiosk 4.0 has also been built on the industry leading graphical interface DevExpress giving us a really shiny, professional and sleek interface. Finally giving us an Interface we can be proud to put on your desktops.

ThinKiosk’s interface has been further improved giving you an Applications tab for Publishing desktops for VMware View, Microsoft Remote Desktop services or Citrix Desktops via ICA file or local applications.

This Applications tab has been modelled after the windows 8 Metro err, I mean Windows 8 UI. This provides a similar look and feel to the new Windows start menu and it really breathes new life into old hardware. With this tab, you can publish shortcuts to VDI Desktops or local applications making it a one stop shop for applications.

You can flick from one tab to another easily, or disable the one you do not wish to use.

It’s all about the customization!

Beauty is in the eye of the beholder right? Agreed!

Themes:

 ThinKiosk 4.0 will ship with over 8 themes and wallpapers, customization of the splash screen, buttons… everything!

The Applications tab can also be completely customized to your tastes:

Lock down:

As with Previous versions of ThinKiosk, every button and object in ThinKiosk can be locked down to exactly what you wish, for example here’s a stripped back browser session:

Or a stripped back application window:

Anyway… Enough about the appearance, Lets talk tech!

Introducing the new ThinKiosk Broker Service and Management console:

The ThinKiosk Broker, Management Console and ThinKiosk clients use an all new ThinKiosk TCP protocol (I never ever, ever want to see a tcp socket again for as long as I live, writing this protocol was a killer!) to allow you to centrally manage, catalog and report on your ThinKiosk devices. The protocol is lightening fast and secure by design.

This new framework will form a long blog post itself, but some quick fire information is below:

• Complete off domain management.
• Auto device registration, just point ThinKiosk at the broker and it will check in and download the default profile.
• Remote Control / Shadowing of end point devices via the console.
• Device Grouping for profiling multiple devices or creating an organisation structure.
• Remote actions (power off, restart, update).
• Device Reporting.
• No Enterprise database software necessary.
• Audit logging.

Unlike other Thin Client protocols and software, ThinKiosk does not accept any inbound connections, in user or system context. Removing the ability to hijack thin clients… which is all too possible with certain vendors!

The console is simple, and quick to navigate:

Installation of the broker takes roughly 5 minutes and is ready to serve your Devices as soon as you configure the default profile.

New Profile Handler:

The ThinKiosk client has received an overhaul and with it we’ve streamlined the profile. ThinKiosk no longer requires group policies or the clunky offline config tool, we have a new profile system based on XML files with a fitting profile editor to match:

No more configuring 5 group policies for one url, the new policy manager is clean, self explanatory, full of new functionality and uses the same interface whether you are using the ThinKiosk management console or modifying the local profile.

If you want to still use group policy to deploy configuration? No problem! just drop the file on the client via group policy preferences!

And the Client!

Lets talk about the 4.0 client.

Supported platforms:

Windows XP – Windows 8

Browser Ahoy!

ThinKiosk is now a fully fledged browser, complete with address bar. If you want to allow your users to browse around, now you can.

Browser improvements:

The ThinKiosk 4.0 browser will:

• Supress scripting errors.
• Allow you to add your sites to the trusted sites via policy.
• Auto tunes the browser for VDI portals.
• Auto circumvent silly SSL untrusted or mismatched errors (great for POC’s *cough* VDI in a Box *cough*)
• ThinKiosk now runs as an Internet explorer executable. No more flicking between iexplore.exe and thinkiosk.exe.

VDI Improvements:

Now to the nuts and bolts!

Local login pass through:

Now that you have the ability to add direct VDI connections. ThinKiosk will handle the log in experience and pass the credentials to the responsible technology:

This integration allows ThinKiosk to better manage the desktop experience and provide your users with a single login pane rather than the recurrent login screens you can experience with Microsoft / Citrix file connections.

These connection files can also be auto launched, to remove that pesky click first thing each day.

Citrix Technologies:

• Log off screen redirection for Web interface, storefront and VDI in a box.
• Log off the web portal when a desktop launches for the above platforms.
• Support for Adding ICA file connections.
• Auto configuration of Single sign on from local pc to remote desktop. (Nightmare previously).
• VDI in a Box auto browser tuning for compatibility.
• Optionally disable the Citrix Desktop viewer (CDviewer.exe).

VMware View:

• Support for publishing multiple pool connections
• Support for publishing multiple direct desktop connections.
• Support for PassThrough.
• Disables Certificate checking by default for quick POC’s.
• Pass through ctrl alt del / Windows + l (more on this later).

Microsoft Remote Desktop Services:

• Support for publishing multiple connections.
• Support for 2012 RDS and VDI.
• SSL Certificate warning suppression.
• Support for login once.

Improved local application handling:

ThinKiosk 4.0 has an improved local application engine, When you add an application to the Applications tab, it will automatically pull in the icon window and you can also specify to launch apps but hide them (think run key entries). If ThinKiosk is restarted via admin task, it’s smart enough to know not to relaunch them.

Environment variables for paths and arguments are fully supported and i’ve also added a variable for 32bit program files paths… I always wondered why Microsoft didn’t do this, but I digress.

Windows secure keystroke blocking and passthrough:

You asked… (and asked and asked and asked and asked). It’s done, with ThinKiosk 4.0 you will be able to block CTRL + Alt + Del, [Windows] + [L] etc.

Pass through of these keystrokes to the remote desktop is available for VMware View already and will be coming shortly after 4.0 for Citrix and Microsoft connections.

Group Policy Lockdown:

By default when you install ThinKiosk 4.0, it will arm the PC with the most restrictive policies via the local group policy engine, disabling access to all admin utilities and even local disks. This lockdown can be tuned or turned off via policy if required.

ThinKiosk performs privileged actions via the ThinKiosk Machine service which installs as part of the installation.

Auto log in account:

ThinKiosk will ship with it’s own user account for fast deployment. This account will be created on the local machine and gives you a quick an easy method to manage local accounts on non domain joined PC.

The accounts password is synchronized with the ThinKiosk unlock password you specify.

This account is completely optional and you can turn it off or substitute it with a domain account of your choice.

ThinKiosk will also manage the Windows Shell replacement policy itself via policy, so no more mucking around with local group policy or registry keys.

ThinKiosk also now encrypts the auto login account using LSA.

Active Setup:

With ThinKiosk as shell, you can now run Active Setup with ThinKiosk’s improved Active Setup Async.

Active setup Async is a utility we have implemented into ThinKiosk that will perform active setup 60% faster than standard Microsoft active setup via a threading and queuing engine, the end result is active setup support ( for example: HDX flash redirection) with a much faster (and prettier)  interface.

Start up Script:

ThinKiosk can now implement the local group policy engines start-up script to allow you to manage off domain PC’s. With the start-up script, you can install software, updates, disable services, uninstall software, delete files, profiles… anything!

The only limitation here is your own imagination or scripting abilities.

If the latter is a concern? worry not, we’ll be creating a scripting library where ThinKiosk enthusiasts can share and collaborate on similar tasks.

Local session control:

ThinKiosk 4.0 offers you the ability to control local volume, printers, screen saver and even background color.

Improved debug logging:

ThinKiosk logs everything, every action, command, hiccup… everything.

If something isn’t quite working as expected, chances are the debugging window will announce in triumphant glory exactly what is broken!

Redundant profile management:

ThinKiosk takes a copy of it’s profile on each check in to an FTP server or Broker server.

In the event of the server being offline ThinKiosk attempts five times to connect before failing back to the local profile allowing your users to continue working without an outage.

If the broker server becomes available again throughout the day, ThinKiosk will check back in to allow management but will not disturb the user.

And so much more!

I’m not going to go on and on, but as you can see… It’s awesome!

Check back in a few weeks for the release as we ready the build.

Just a quick note to point out I’ve added ThinKiosk 3.3 to the download list.

ThinKiosk 3.3 is a big collection of bug fixes for application launching, process launcher etc and has been available to customers for a number of weeks without any issues. If you are facing any niggles please do drop by the download page and grab a copy. All the new functionality I’ve been leaking on twitter will be available in 4.0.

As for ThinKiosk 4.0, it’s a massive overhaul, includes tons of new functionality and Beta testing while very positive, is taking alot of time to iron out the creases. Expect a release before July.

I’m absolutely delighted to announce that ThinKiosk has made it’s way into the Norskale User Environment
Management (UEM) stack: VUEM (or Virtuall User Environment Management). It will be appearing as an additional module named “Transformer”. This has been in the pipeline for some time now and it’s great to be finally ready to talk about it!

About Norskale, in my opinion:

For those of you who may be unfamiliar, Norskale are the new up and coming stars in the UEM market. Norskale is the technical brain child of Citrix CTP, community expert and all round nice guy Pierre Marmignon. Pierre brings over a decade of user virtualisation experience to the table. Pierre is acutely aware of the failings in the UEM/ Remote Desktop market and is using these to great effect, creating a lean, extremely fast and easy to use end user management solution at a good price to boot.

Pierre is backed by extremely clever development / marketing and management teams and with their help, Norskale is making serious waves, just in their first year, shaking a market segment that was somewhat dormant for many years.

How did this transpire?

Pierre reached out to me late last year about a potential partnership between ThinKiosk and Norskale. Funnily enough Pierre reviewed the code of ThinKiosk 1.0 and as quite a new comer to this community,Pierre has always been an idol of mine in the Citrix community. It was because of Pierre and a handful of other guys in this community I decided to start blogging and release free tools.

With all this in mind, when Pierre reached out, I jumped (backflipped) at the possibility to work with him!

And how has it turned out?

I’ve worked closely with the Norskale development team as they completely refactored ThinKiosk’s code to make their own VUEM integrated version that would suit their application framework and leverage all the functionalities they were already providing.I’ve been blown away with the results. Norskale VUEM is already a fast, feature rich and clever offering, but with a ThinKiosk style Workstation to ThinKiosk conversion they now can offer a unique end to end solution for user virtualisation in VDI and Server Based Computing (SBC).

Keeping simplicity as key, it’s a matter of a checkbox to convert a pc to a Thin Client using Norskale, and just as simple to roll back completely should you need to. The Norskale solution has been written, debugged and signed off by a team of wizard developers and I couldn’t be more proud of the results.

So now you can instantly roll out any VDI or hosted desktop initiative in a snap and without heavy investment in traditional thin client hardware. And you are getting the best level of performance and manageability one can think off.

And the nitty gritty?

With Transformer, Norskale now have a ThinKiosk style solution as an add on to VUEM, their UEM solution which has all the features of the ThinKiosk product (and much more) to offer to it’s customers.

Norskale’s Transformer will be maintained and evolving as a separate product entirely. But as part of our partnership, all improvements, features and solutions I add to the ThinKiosk solution will be available to Norskale for use in their own product. With this they get all of the benefits of their enterprise solution, along with the innovations ThinKiosk customers and aficionado’s are driving.

What about ThinKiosk?

ThinKiosk will remain free to use, will continue to be developed… I have a massive release coming out shortly. Worry not!

ThinKiosk is not going anywhere and if you are a Norskale customer, or considering becoming one you will get the benefit of having their end to end solution and any features I add to the ThinKiosk product, all with Enterprise grade support and scalability. It really is a win/win solution for all involved.

Going forward.

Norskale has a big, big future. It was an absolute blast to meet their development team and work with them on this project, I’m honestly looking forward to working with them again on new idea’s and solutions.

Got an idea of your own?

If like me you had a unique idea but struggled to see the bigger picture go speak to Norskale, NOW. They will help you leverage your idea, bring it to market and help you add to a game changing solution. They understand the space, the challenges of creating a product on our own and then trying to monetize it.

Hi Guys and Gals. I’m currently fighting the good fight with Microsoft support and require your help and backing in order to close down a long standing bug in the Windows Explorer Shell.

As you are all aware, hiding the c: drive and restricting access has been a utility we use frequently in shared computing and VDI environments. Restricting this functionality removes views of the shared drive from users and adds a layer of security and complexity* to ensure the users in question have access to only what they need in order to do their jobs day to day.

*I’m not looking to argue the merit of doing this either, it really depends on the business case or environment to dictate whether this option is set. I’m NOT saying it should be done in every case.

We all know it’s not fool proof, there are certain ways for users to circumvent this layer and I particularly don’t want to discuss them here to give potential devious users a landing page for idea’s!

The problem:

Prior to windows Vista, when you hide the c: drive and an application requests access to a c: drive folder, be it from an “open save dialog” or otherwise, Windows detects this event knows that the folder is restricted and merely redirects them to the desktop which they can see then browse to where they wish to open or save a document. This has worked fine to memory since windows server 2000.

But with the changes to Windows Vista’s windows explorer, repeating the above steps will result in the following annoying, unnecessary and interrupting error message “This operation has been cancelled due to.. bla bla blah”:

This issue can be easily recreated, simply hide and restrict the c: drive, then click start > run > browse… bang.

The more annoying problem here, is after the error message, windows simply redirects back to visible folder. In most cases this is the documents library. So the error message is simply poping up then reverting to the functionality seen in previous operating systems.

So to review:

• Issue introduced in Vista / 2008 and above.
• error message displays.
• Previous redirect functionality is still there and occurs after ok is pressed.

To Microsoft!

Being a pedantic individual, along with my colleague we brought this to Microsoft support and somehow lost months in the conversation as follows:

1. Microsoft then redirected us to RES Software.
2. Who (although very nice about it) sent us back to Microsoft.
3. At which point I got involved.

Now with the correct audience and suitable severity, this problem has been identified as “introduced in Windows Vista” as an “Added Security feature“. How an annoying pop up box, masking previous functionality is a security feature is anyones guess, but it’s bloody annoying…

We have raised this as a bug and have requested Microsoft to fix it. The change in question was deemed as large change or substantial change due to WIndows explorer being used by all of the operating systems and basically told, without significant backing, this change wont be implemented.

Bureaucracy and broken policies, yes but that doesn’t help my customer.

Here’s where I need you:

In order to bolster this change and fix an issue in our beloved operating systems for Server Based Computing and VDI environments I need to hear from you and your customers to confirm they have had this issue, or currently face the issue and wish for a fix.

• If you are a customer and suffer this issue, email me.
• If you are a consultant and have customers with this issue, email me.
• If you or your customer have enterprise support with Microsoft, I ESPECIALLY want to hear from you.

What’s in it for you?

Microsoft have provided us a work around, as a process that watches window messages and suppresses this dialog box when it occurs. If you get in touch, I’ll recompile this application with Microsofts permission and pass it on to you for use in your environment while we get “The Man” to fix it!

This fix is a bit of hack, as it’s scraping window messages but it’s light weight and scalable. Use this process for now and I’ll provide you with updates on a fix as and when I get them.

How do you contact me?

Please drop me and email on andrew{at}andrewmorgan{dot}ie with the following information:

• Customer name:
• Affected users:
• Has enterprise support: (yes/no)

Once I have that information, I’ll send you back an executable via dropbox and keep you updated on the call process. This information is merely going to be fed straight to Microsoft with my personal guarantee of confidentiality. No funny business.

If you can’t share customer information, but have suffered this issue in the past, no problem! Please comment on this blog post the number of seats that were affected and roughly how many times you’ve seen it.

That’s it!

Thanks for entertaining my request for help and hopefully you too want to get this issue fixed as much as I.

I wanted to use the PendingFileRenameOperations registry key to instruct windows to copy a file during the boot process.

The issue with this key and behavior is that in order to tell windows to delete a file, the next line to the source file must be blank… if you manually try to add a blank line to regedit you receive the following error!

I needed an application to move a file during the boot process of windows before the service or handle held the file I wanted to replace open. I decided to write a new tool called MoveOnBoot.exe.

MoveOnBoot leverages the PendingFileRenameOperations registry key and the MoveFileEx Api to move the file on boot simply and easily.

Move on boot does the following:

1. Adds the copy jobs to the PendingFileRename key you specify.
2. Copies the new file into the target directory with an  _newer file extension.
3. Optional: instructs windows to copy the current file to a _old extension
4. instructs windows to replace the target file with the _Newer file.

How to use it:

Simply select the source and destination files as below:

Optionally choose to backup the target file during the operation with the check box above.

Once you have added all the files you need to replace, you can check the queue by going to file > view pending operations:

And that’s it! restart the device and let windows do the hard work.

Optionally, if you chose to backup the file as part of the operation, you will find an _old file in the target directory as below:

Download:

Stand alone Binary.

Source code.

Support information:

• Requires Administrator privileges.
• Requires .net framework 2.0 or greater.

Not happy with the Irk, and still on my app developing buzz, i decided to write SBC Printers:

SBC-Printers is a simple little .net 4 application, leveraging WMI for printer enumeration and control.Because SBC Printers is an executable, it can published as a XenApp application. Sbc Printers can also be installed as the default printers interface on the start menu:

So really your users won’t know the difference or care for that matter!

SBC-Printers also comes with securable options for adding or deleting local printers:

The display of add or delete can be controlled via the settings file in the installation directory:

Installation:

1. Download the following MSI
2. Install the MSI to the default directory.

To restrict the standard printers dialog from users, but leaving it accessible to administrators:

• Browse to c:\program files (x86)\SBC-Printers\bin

• run the powershell script below, make sure to run it as an administrator!

That’s it, once the Powershell script runs. it removes the users access to the registry classes giving them access to the standard devices and printers interface. Which means we’re now ready to provision SBC-Printers to replace it.

Provisioning the replacement to the user:

Now just import the userkey.reg into the users profile on login, you can do this via your user profile manager of choice, or use Group Policy preferences.

That’s it!

As you can see I haven’t streamlined the install process too much, this is mostly down to the simplicity of the tool. If you like SBC-Printers but would like a better installer, just drop me a comment below.

Roll back:

if you need to restore the standard interface, uninstall SBC-Printers then add the (local computer\users) group back to the following registry keys ACL:

•  HKCR\software\classes\CLSID\{A8A91A66-3A7D-4424-8D24-04E180695C7A}
• HKCR\software\Wow6432Node\CLSID\{A8A91A66-3A7D-4424-8D24-04E180695C7A}

• Height: 2048
• Width: 1056

So if you want to replace this file, go find your replacement picture and ensure your picture is of a similar enough size.

Once you have a png file with similar enough dimensions, open the finder application, open the applications folder and right click the Citrix Receiver app, choose “Show Package Contents”.

Browse down to: contents > resources





In this folder, you will find a file “backgroundImage_big_b.png”, before you start, rename this file to back it up.

Now simply copy your replacement file into this folder, using the above name:





And that’s it! You’ve now got a lovely custom Citrix Receiver:





PS: I wouldn’t try to do this with windows, the file is an embedded resource and would require resource hacker to change the file.

Improvements:

• You can now tell ThinKiosk to auto launch your custom tools when you start the machine.
• You can now use variables in the custom title if you wish.
• Windows Remote Desktop Services can now do “end of session” options and will handle brokered connections.
• The printers interface has been overhauled to be more friendly.
• The language selected by the last user is now retained, this can be disabled via GPO / Offline Config Tool.

Bug fixes:

• Time issue where zero’s were placed after the number rather than before in the Offline Config tool.
• Custom tools handling of spaces and quotes (“) improved.
• Bug in Norwegian language appearance fixed.

In other news:

A support forum has been setup for ThinKiosk, you can find it here.

And it’s still free!

ThinKiosk development has taken quite some time and it takes time to support you via email. If you use ThinKiosk in your environment or appreciate the savings its made for you, please consider making a donation or paying for enterprise support to help me keep this project alive… I would really appreciate it as it will allow me to invest in better development tools to make the product look and feel even better!

An Issue began two weeks ago intermittently in a XenApp 4.5 farm used for hosted desktops, intermittently NonPagedPool bytes would shoot through the roof, the event logs would become inundated with event 333 errors and the servers would lock up completely.

The running sessions could no longer open new applications, performance was extremely poor and RDP’ing to the XenApp server would result in an RPC error. Disconnecting the sessions remotely would also result in an RPC error or TSAdmin was completely incapable of connecting to the server. We had no choice but to dump the servers using NMI and pray for a crash dump.

No changes had been made to the environment in a number of weeks and the last change to the environment was a “Citrix Ready” printer driver from Lexmark. As the days progressed the issue got worse and worse with more servers going offline day by day. Although we did initially catch a number of crash dumps, we hit a bad run of luck with most of them being corrupt on restart.

By day six, 9 servers went offline throughout the day, I was pulled in to assist resolve this massive issue.

I fired up the windows debugging tools and managed to get a look at a crash dump fresh from a locked up server.

Using !vm i pulled the virtual memory statistics at the point of the crash:





So we had a serious non paged pool leak as we suspected, but what exactly was chewing up all that nonpaged?

Running !poolused 2, i was able to drill down into the drivers using nonpagedpool and view which driver tag was using the largest amount of the pool as below:





reviewing the list, i was immediately alarmed by the amount of ram in use by the “Ica” pool tag. Having reviewed 100′s of memory dumps I had never seen the Ica pool tag listed in the top 20, never mind using 99721328 bytes (~95mb).

The Ica pool tag is fairly obvious as to who owns it, but just to be on the safe side and to drill down to the owning driver, I ran the following command on the drivers folder to find references to the “Ica” pool tag.



findstr /m /l Ica *.sys



So we got quite a few hits off the Ica pool tag. Quite a number of the above drivers are Microsoft’s, which is not suprising in the grand scheme of things as we all know the origination of the RDP protocol.

So with a little more information to hand, I set about googling this chain of events to see if it’s either documented, or hotfixed. A search yielded quite alot of articles including a private hotfix and a Rollup pack.

Drilling down into the technotes to see if I could find a potential cause for this issue, I was left a little wanting with the lack of information available:

Servers with Hotfix Rollup Pack 6 installed can run out of non-paged pool memory in the ICA Pool tag. The issue occurs when users log on to the server and its frequency increases with the amount of time since the most recent restart and the number of logons since the restart.

What irked me here, was the lack of information and the fact that these servers had been running HFRP 6 for roughly 18 months with no issues similar to this.



Why all of a sudden are we losing servers all over the place to such an issue?



I dug further into the hotfix notes with help from my good friend and all round cool Citrite James Denne, the hotfix specifically noted:

When a server is in low memory condition the <Redacted>() spins in an infinite loop by constantly allocating an OUTBUF and trying to write it on the stack. This problem happen when system is going in and out in low memory condition.

So there’s a great explanation of the issue from the horses mouth, but again there was a niggling problem in the back of my head…



These servers weren’t spinning in and out of low memory, our pool usage reporting would have caught this?



I was satisfied to see a hotfix was available, but in the back of my head I was concerned about the change that may have caused this issue, it’s still unclear what is causing this low memory condition to spin the infinite loop and why we couldn’t see the low memory scenario before it happens. Being an massive issue, we had to make a quick turn around here. We had a choice of going to HFRP 7 or using the private hotfix available. I chose the private hotfix, for two reasons:

• Mass Deploying a roll up pack to fix one problem is like tapping a nail in with a sledge hammer.
• My experience with HotFix Rollup Packs is they fix your issues, but introduce at least one new one.

We took all the servers offline for emergency maintenance that night and cautiously waited for the morning to come and see if our issue was resolved.



and so we patiently waited…



Once hotfixed and rebooted, we arrived at the office early to watch as the user sessions began to filter in to the Farm. All was quiet for the first hour or so, but then the phones started.

once the user load hit 15-16 users per XenApp session, a number of servers began to again log a number of eventlog 333′s as below:






Dammit.



Frantically we connected to the console of a server, to check the paged pool states but again no alerts on pagepool size? as below the ICA pool tag was nowhere to be seen:





And the ica tag was at a much more respectable / expected value as below:





Our next clue came in the form of the following, when users were logging in they were getting the following error:






So we’ve fixed our Ica memory leak, now what the hell is happening?



If memory usage for the pools are ok but we’re still getting errors about flushing to the registry, and now new user profiles can’t load their profiles, my hunch was there had to be something wrong with the registry hives…

I used command prompt to open the “Documents and Settings” folder and ran the following command:

dir /s /a ntuser.dat

With a quick glance, i found the following:





The “Citrix Print Manager Service” user account had a registry hive of over 50mb? What in the name of superman is hiding in that registry hive?

To rectify the issue immediately, we stopped the above print manager service and forced the hive to be unloaded with delprof. Once we had done this, the user profiles began to load again on each affected server. But we’re now unable to use client redirected printing.



To regedit!



I mounted the registry of a profile that had failed to delete and drilled down to see what all the fuss was about. As this was  now firmly in the printing land, I went looking for keys to match the Lexmark driver change from a number of weeks ago.

What I found was extremely interesting, for each client redirected printer ever mapped with the Lexmark driver, we had an entry under both the PCLPlugin and PSPlugIn keys:





Although this was a really questionable practice from lexmark, I examined the keys under each entry for the PclPlugin key and they consisted of just a few small binary files of which were no more than a KB or two.

Upon looking at the same keys under PSPlugin, I found a new key, called GDL. This GDL key was absolutely massive and there was one for each and every time a client printer had been redirected using the Lexmark V2 driver.





I exported both both the users hive, and the psplugin key to text and the comparison is below:





The GDL key itself was over 3mb per redirected printer!?!?:



So there we have it.



The route cause was as follows:

This Lexmark driver has a weird tendency to store an unbelievable amount of crap in the registry for users.

The Citrix print manager service also suffers this faith when it maps a redirected printer.

As more and more users were testing in production (GRRRR!) / beginning to use a new printing solution on a customer site, this registry file began to grow and grow ultimately flooding the maximum registry size of 30% of the paged pool ram.

As the registry hive size was growing out of control, the Ica driver ran into a low memory situation and ultimately caused the infinite loop.

The Ica loop and nonpaged saturation was masking the printer driver bloat in the registry.

As the days went on, more and more servers began to saturate the Maximum registry size and go offline.

Corrective actions:

• Enforce a policy to not allow native drivers, in any way, shape or form when redirecting printers where possible.
• Obtain the latest driver from Lexmark is you have lexmark printers.
• Give lexmark an earful for not testing their drivers.

Lessons Learned:

• Don’t test things in production.
• Don’t trust a vendor with “Citrix Ready”, it’s up to them to test these things and they regularly don’t.
• Create a monitor for registry size (perfmon > system > % Registry quota in use)
• install the debugging tools on the XenApp 4.5 servers as this issue is going to become more prevalent. *

* This isn’t going to get any better.

As Vendors move further and further towards 64 bit architectures they can and will forget about the extremely restrictive memory sizes available in 32 bit versions of windows, 64bit windows has so much memory available for the pools they can be as sloppy as they want without much concern.

Server 2003, Windows XP and XenApp 4.5′s death bells are knelling and have been for some time.

You are going to see pagepool’s floods and other such nasties more and more in the coming months before you finally decommission your old server 2003 environment. My advice to you is to:

• get very comfortable with the following tools:
  • PoolMon.
  • Process explorer.
  • Windows debugging tools.
• Have a good read of the following article: 333, read it, read it again.
• Never be afraid to have a look at a dump file yourself.
• Throw an issue at every vendor possible during troubleshooting, it’s in their interest to prove it’s not their software at fault.
• Understand your pagepool sizes and limitations.
• Never trust a printer driver.
• Never, ever, ever trust a Vendor to behave accordingly or follow the Citrix Ready standards.
• If you absolutely, positively need to run something in server 2003 or XP, consider using XenDesktop hosted apps to isolate the problem to a singular kernel away from the bulk of your task workers.

Having been in this situation recently, I needed to audit and report on the types of files open on the file server, my hunch was a certain select number of users were running applications (like *gulp* lotus notes) from the network share.

Disappointed with the powershell scripts on the interwebs, I decided to write my own function to perform this task:

function get-openfiles{
param(
    $computername=@($env:computername),
    $verbose=$false)
    $collection = @()
foreach ($computer in $computername){
    $netfile = [ADSI]"WinNT://$computer/LanmanServer"

        $netfile.Invoke("Resources") | foreach {
            try{
                $collection += New-Object PsObject -Property @{
        		  Id = $_.GetType().InvokeMember("Name", 'GetProperty', $null, $_, $null)
        		  itemPath = $_.GetType().InvokeMember("Path", 'GetProperty', $null, $_, $null)
        		  UserName = $_.GetType().InvokeMember("User", 'GetProperty', $null, $_, $null)
        		  LockCount = $_.GetType().InvokeMember("LockCount", 'GetProperty', $null, $_, $null)
        		  Server = $computer
        		}
            }
            catch{
                if ($verbose){write-warning $error[0]}
            }
        }
    }
    Return $collection
}

The function above (get-openfiles) has been written to accept an array of servers to the command line and it will return the following items:

• The ID of the open file.

• The server it’s open from.

• The username who has the file open.

• The amount of locks the file has.

A couple of quick examples for using this command are below:


Retrieving open files from server1:






get-openfiles -computername server1 | select server,itempath,lockcount



Retrieve a count of open files that end with the nsf file type (Lotus Notes):






(get-open files -computername server1,server2 | ? {$_.itempath -like "*.nsf*"}).count()



Retrieve a report of total open files on a number of file servers:



get-openfiles -computername server1,server2,server3,server4,server5 | group -property server