Category Archives: Group Policy

The Curious Case of High CPU usage on Login with Citrix XenApp.

This was such a weird, wonderful and perfect storm issue I felt I had to post it.

Recently while configuring a XenApp 6.5 hosted desktop environment we experienced a weird CPU spike across all XenApp sessions on a server. Intermittently throughout the day, the XenApp’s CPU would max out, and as the load increased on the XenApp servers the timeframe in which the CPU would max out would get longer and longer.

From Initial troubleshooting we found that after 7 users logged into the XenApp server, the next user login would cause the server to hang.

 

To Process explorer!

 

During this hang, using process explorer we found an instance of taskhost.exe for each logged in user running at this time:

 

 

As most of you will already know, taskhost.exe is the process handler for scheduled tasks, and as most of you will also know scheduled tasks have changed in a big way since server 2003.

In Server 2008 and upwards scheduled tasks are no longer limited to times or schedules, scheduled tasks can now operate on triggers or events too.

The user login process in our case seemed to be triggering an event across all sessions that was consuming all the CPU, but which task runs on every login?

 

To Powershell!

 

As I ventured further down the rabbits hole on this one, I resorted to my good friend PowerShell to catch a user login and see which processes or “Scheduled Tasks” were running at this time. As the CPU was maxing out during this process, it was quite easy to catch first time.

Below is the command I used to catch the running tasks during the CPU spike:

[sourcecode language=”PowerShell”]
$ts = New-Object -ComObject Schedule.Service
$ts.Connect($env:computername)
$ts.GetRunningTasks(0)
[/sourcecode]

and the output of the above command gave us the following hint:

 

 

To Server Manager!

 

Taking the above path as an indicator, I launched Windows Server Manager and browsed to Configuration > task scheduler > Task Scheduler Library > Microsoft > Windows > CertificateServicesClient:

Under this task category, we can see 3 tasks.

 

 

As above, its the UserTask that seems to be running, so lets have a look at this. Below is a list of triggers configured for this task. As you can see, the two interesting triggers are the “On an event” & “At Logon”.

 

 

Looking at the history tab, we can see that during a user login, both events are triggered as below. The Second trigger seems to be related to the event log which turned out to be not related as Microsoft supress this task from running twice it seems by assigning a zero value GUID to the task.

 

 

The first trigger however is the money shot, As you can see below this user log in caused this event to trigger, which then triggered in a cascading effect in all user sessions:

 

 

So with the culprit at hand and more troubleshooting, we found that Interestingly, if a user disconnected and reconnected, this issue did not occur. And even more interesting again, if a stale user profile belonging to the user was still present on the server, this task did not trigger.

As with most of my implementations I chose to use mandatory profiles with this customer, with some sleuthing it turns out this event was triggering for the user login only if this was a brand new user profile creation (from mandatory) and only in this event was further tasks being spawned in the user sessions.

So to review, every day, when users were creating “new” or “clean” profiles from the mandatory profile this event would trigger for them and in turn for every other logged in user.

Going to the lab, and talking to a Microsoft support representative, it turns out this turn of events is by design, frustratingly Microsoft stopped before they confirmed that this event triggering in all users sessions is a bug, but agreed it was unnecessary. The confusing thing to arrise out of all of this was:

If this is by design, why is only this customer suffering this issue and not every Citrix XenApp environment with Mandatory profiles?

 

To Group Policy!

 

Looking further into certificate settings in the customer environment, It turns out that, as part of the default, enforced domain policy, the customer is assigning a large amount of trusted root certificates:

 

 

Now this is not the way Microsoft recommend you deploy Root Certificates, but due to a corruption in Certificate Services and a Microsoft representative proposing this as a work around. This is how the customer was deploying their trusted roots.

But this is a computer policy? Why is this hampering user logins?

Yep, you guessed it, Group Policy loop back processing!

 

 

Group policy Loop back processing was causing this computer policy to reapply on each login for the user.

So in review, this issue was caused by:

  • Task scheduler in windows triggering events in user sessions on every login to a server.
  • Utilising Mandatory profiles with Windows RDS.
  • The customer storing a large number of Certificates in Group policy.
  • Loop back processing enabled without full consideration of the policies being applied.

You took me this far, how did you resolve it?

 

Well if you’re curious to know how we got around this issue, read below:

1: Once I realised it was taskhost.exe that was causing the CPU spikes, I utilised an application i wrote ThreadLocker, to restrict taskhost.exe to:

  • two cores
  • its process priority to idle

This resulted in the tasks taking much longer to complete but the user sessions were uninterrupted during the process.

This bought us precious time to troubleshoot as this issue was in a production environment.

2: Once we realized this issue was related to the certificate services, we disabled the client scheduled task temporarily while we devised a new solution for active directory. We could not disable Loop Back processing due to its dependency in the environment.

3: The customer moved the certificates to a non enforced domain policy and we restricted this policy from propagating to the XenApp servers. We then re-enabled the client task and remove the rule from ThreadLocker as it was no longer needed.

Controlling the creation of Libraries in Windows 7 / Server 2008 R2.

Following on from my previous post about libraries, I have found you can actually control library creation, but there is a two fairly large caveats I’ll cover later in this post.

To Block Library creation, you can create a user group policy blocking the known folder ID. You may argue a login script can simply delete the unwanted libraries, and you would be right, but a shell context menu exists to restore these libraries on user request. The method below ensure’s they are never created.

For every default windows directory, these directories have known folder names and GUID’s. A great reference site for these GUIDS can be found here:

http://msdn.microsoft.com/en-us/library/dd378457(VS.85).aspx

In our case, we’re interested in the following five Known Folders:

FOLDERID_DocumentsLibrary   GUID{7B0DB17D-9CD2-4A93-9733-46CC89022E7C}
FOLDERID_MusicLibrary             GUID{2112AB0A-C86A-4FFE-A368-0DE96E47012E}
FOLDERID_PicturesLibrary         GUID{A990AE9F-A03B-4E80-94BC-9912D7504104}
FOLDERID_VideosLibrary            GUID{491E922F-5643-4AF4-A7EB-4E7A138D8174}

and the little known:

FOLDERID_RecordedTVLibrary  GUID{1A6FDBA2-F42D-4358-A798-B74D745926C5}

Once we know which folders we wish to block, open group policy and navigate to the following policy:

User Configuration > Policies > Administrative Templates > Windows Components > Windows Explorer.

In this section, you will find a Disable Known Folders Setting.

Enable the policy and click show, here you can configure the libraries you wish to block:

(below I’m blocking the creation of Video’s and Music.)

Configuring the value above will leave you with libraries as so:

Caveat’s:

It’s a once off thing… Blocking the creation of a library will only take effect on the first login, aka the profile creation. There is no microsoft solution available to control these libraries after the fact. You could make do with login scripts, but its messy.

It’s not the size, its the contents that matter… You cannot control the cotents of libraries, i.e. you cannot block the link to the shared folders libraries. This is really silly, as you would assume that were you to block the known folder PublicDocuments aka {ED4824AF-DCE4-45A8-81E2-FC7965083634}. This would stop it being created in the profile on first load, you’d be very wrong, blocking this known folder causes the library to not be created. Not sure who thought that was a good idea, but I digress.

So that’s it in a nutshell, for more information on locking down the server 2008 r2 profile check in later or follow me on twitter @andyjmorgan.

Problems using Group Policy to set a mandatory profile…

Here’s a weird one i came across this week while configuring a mandatory profile, I was using my local laptop to configure group policy over the domain and could set the mandatory profile, but could not set the “Do not append the user name to the profile path” as it was missing from the group policy object!

After much head scratching we discovered an extra registry setting embedded in the group policy:

HKEY_LOCAL_MACHINESOFTWAREPoliciesMicrosoftWindows NTTerminal
ServicesWFDontAppendUserNameToProfile dword = 1.

Still none the wiser to the issue we found the following article http://support.microsoft.com/kb/908011. With this we put two and two together, and logged into a 2003 domain controller (sp2) and low and behold the policy setting was available.

It seems the group policy were updated in service pack 2 to reflect an issue when trying to configure mandatory profiles, I do aim to copy the policies to my laptop and see if i can update them to reflect the change but I’ve been to lazy so far :)

Update: Well i got around to testing the policy copy, if you copy the .adm files from the domain controller to your local machine with the problem, it will resolve the issue.

Heres an example of scripting the change:

pushd domaincontrollerc$windowsinf
copy *.adm %windir%inf
popd

That’s it. reload gpedit.msc and you should see hte entries correctly.

how do i create custom .adm / group policy files?

 

Update: With thanks to some great help and troubleshooting from Steven we have resolved the line 46 “Categor” error. In order for the adm to parse the ending y in this file an additional two blank lines or “carriage returns” are necessary at the base of the adm file. The download file has been updated, Thanks again Steven.

A .adm file, is a group policy file that specifies policies outside of Microsoft’s default options. Basically they are policies you can put in place that Microsoft in their infinite wisdom forgot to put in before launch.

I had a situation recently where we have external users coming into our network, and using our CAG’s to access the the citrix environment. Once in there they needed access to an internal webpage that we published with internet explorer. The problem therein lied that these users could browse the local lan for resources with the address bar and many other wonderful utilities Microsoft put into internet explorer but failed to lock down efficiently.

All i really cared about (and for the interest of this post) was locking down the address bar in Internet Explorer 6.1. Nowhere could i find an option to do this, and i was getting nowhere fast. Searching internet explorer did bring back a few “helpful” articles on technet that i just couldnt understand, and i did find a piece of software that used to do it for free, until microsoft bought the company, stole its code for server 2008 and stopped people using or downloading the application. nice one microsoft…

I have attached the policy settings and ADM files for reference on how to lock down internet explorer 6 completely, hopefully i will save somebody else 7 hours of their time.

Continue reading