Category Archives: Active Directory

The Curious Case of High CPU usage on Login with Citrix XenApp.

This was such a weird, wonderful and perfect storm issue I felt I had to post it.

Recently while configuring a XenApp 6.5 hosted desktop environment we experienced a weird CPU spike across all XenApp sessions on a server. Intermittently throughout the day, the XenApp’s CPU would max out, and as the load increased on the XenApp servers the timeframe in which the CPU would max out would get longer and longer.

From Initial troubleshooting we found that after 7 users logged into the XenApp server, the next user login would cause the server to hang.

 

To Process explorer!

 

During this hang, using process explorer we found an instance of taskhost.exe for each logged in user running at this time:

 

 

As most of you will already know, taskhost.exe is the process handler for scheduled tasks, and as most of you will also know scheduled tasks have changed in a big way since server 2003.

In Server 2008 and upwards scheduled tasks are no longer limited to times or schedules, scheduled tasks can now operate on triggers or events too.

The user login process in our case seemed to be triggering an event across all sessions that was consuming all the CPU, but which task runs on every login?

 

To Powershell!

 

As I ventured further down the rabbits hole on this one, I resorted to my good friend PowerShell to catch a user login and see which processes or “Scheduled Tasks” were running at this time. As the CPU was maxing out during this process, it was quite easy to catch first time.

Below is the command I used to catch the running tasks during the CPU spike:

[sourcecode language=”PowerShell”]
$ts = New-Object -ComObject Schedule.Service
$ts.Connect($env:computername)
$ts.GetRunningTasks(0)
[/sourcecode]

and the output of the above command gave us the following hint:

 

 

To Server Manager!

 

Taking the above path as an indicator, I launched Windows Server Manager and browsed to Configuration > task scheduler > Task Scheduler Library > Microsoft > Windows > CertificateServicesClient:

Under this task category, we can see 3 tasks.

 

 

As above, its the UserTask that seems to be running, so lets have a look at this. Below is a list of triggers configured for this task. As you can see, the two interesting triggers are the “On an event” & “At Logon”.

 

 

Looking at the history tab, we can see that during a user login, both events are triggered as below. The Second trigger seems to be related to the event log which turned out to be not related as Microsoft supress this task from running twice it seems by assigning a zero value GUID to the task.

 

 

The first trigger however is the money shot, As you can see below this user log in caused this event to trigger, which then triggered in a cascading effect in all user sessions:

 

 

So with the culprit at hand and more troubleshooting, we found that Interestingly, if a user disconnected and reconnected, this issue did not occur. And even more interesting again, if a stale user profile belonging to the user was still present on the server, this task did not trigger.

As with most of my implementations I chose to use mandatory profiles with this customer, with some sleuthing it turns out this event was triggering for the user login only if this was a brand new user profile creation (from mandatory) and only in this event was further tasks being spawned in the user sessions.

So to review, every day, when users were creating “new” or “clean” profiles from the mandatory profile this event would trigger for them and in turn for every other logged in user.

Going to the lab, and talking to a Microsoft support representative, it turns out this turn of events is by design, frustratingly Microsoft stopped before they confirmed that this event triggering in all users sessions is a bug, but agreed it was unnecessary. The confusing thing to arrise out of all of this was:

If this is by design, why is only this customer suffering this issue and not every Citrix XenApp environment with Mandatory profiles?

 

To Group Policy!

 

Looking further into certificate settings in the customer environment, It turns out that, as part of the default, enforced domain policy, the customer is assigning a large amount of trusted root certificates:

 

 

Now this is not the way Microsoft recommend you deploy Root Certificates, but due to a corruption in Certificate Services and a Microsoft representative proposing this as a work around. This is how the customer was deploying their trusted roots.

But this is a computer policy? Why is this hampering user logins?

Yep, you guessed it, Group Policy loop back processing!

 

 

Group policy Loop back processing was causing this computer policy to reapply on each login for the user.

So in review, this issue was caused by:

  • Task scheduler in windows triggering events in user sessions on every login to a server.
  • Utilising Mandatory profiles with Windows RDS.
  • The customer storing a large number of Certificates in Group policy.
  • Loop back processing enabled without full consideration of the policies being applied.

You took me this far, how did you resolve it?

 

Well if you’re curious to know how we got around this issue, read below:

1: Once I realised it was taskhost.exe that was causing the CPU spikes, I utilised an application i wrote ThreadLocker, to restrict taskhost.exe to:

  • two cores
  • its process priority to idle

This resulted in the tasks taking much longer to complete but the user sessions were uninterrupted during the process.

This bought us precious time to troubleshoot as this issue was in a production environment.

2: Once we realized this issue was related to the certificate services, we disabled the client scheduled task temporarily while we devised a new solution for active directory. We could not disable Loop Back processing due to its dependency in the environment.

3: The customer moved the certificates to a non enforced domain policy and we restricted this policy from propagating to the XenApp servers. We then re-enabled the client task and remove the rule from ThreadLocker as it was no longer needed.

Using my Citrix Edgesight Powershell module with Active directory OU’s.

I received a request on twitter late last night and it was an interesting one. The person in question wanted to use my current edgesight module to import users from active directory into the static Citrix Edgesight groups, but instead of group membership in Active Directory, they wanted to use Active Directory Organisational Units.

All the information on how to use the module is included in the previous post, so I wont re-invent the wheel. Have a read of the previous post for any caveats or pre-emptive misunderstandings.

Below are two code snippets to use OU membership with either the Quest or Microsoft cmdlets for active directory, just modify the OU Path below, I’ve tried to include a long example to ensure there’s no confusion.

 Quest Active directory Snap-in:

[sourcecode language=”Powershell”]
#Quest Active directory module
import-module "C:citrix.edgesight.cmdlets.psm1"
add-pssnapin Quest.ActiveRoles.ADManagement
$ADOU=’domain.domain.com/Country/Users/advanced/Helpdesk’
$esgroupid=20

#clear the group before import
clear-esgroupmembers -groupid $esgroupid

#get users from group, then import them into edgesight
foreach ($user in get-QADUser -SearchRoot $ADOU -SizeLimit 0){
$prid = get-ESUserPrid $user.logonname
if ($prid -NE $null){
Add-ESGroupMember -groupid $ESgroupid -prid $prid
}
}#end For
[/sourcecode]

Microsoft Active directory module:

 

[sourcecode language=”Powershell”]
#Microsoft active directory module
import-module "C:citrix.edgesight.cmdlets.psm1"
import-module activedirectory
$ADOU="OU=helpdesk,OU=advanced,OU=Users,OU=Country,DC=domain,DC=domain,DC=com"
$esgroupid=20

#clear the group before import
clear-esgroupmembers -groupid $esgroupid

#get users from group, then import them into edgesight
foreach ($user in get-ADUser -filter * -searchbase $ADOU){
$prid = get-ESUserPrid $user.samaccountname
if ($prid -NE $null){
Add-ESGroupMember -groupid $ESgroupid -prid $prid
}
}#end For
[/sourcecode]

Move objects in active directory which have been inactive for x days.

This is just a quick script I was asked for assistance with recently. The person in question wanted to move all computers and users to defined ou’s when they were inactive for 90 days.

The script is fairly self explanatory but quite scary if you get it wrong, for that reason I’ve included the -whatif parameter to show you what will happen if you overzealously just copy and paste the code. Once you are happy it works, remove the whatif parameters.

This script relies on the powershell module for active directory, you can see if its installed as below:

I’m also aware this code is quite inefficient by searching twice, but it was the cleanest appearance I could muster to ensure the end user understands what is happening.

[sourcecode language=”Powershell”]

#Load the required Snapins
if (!(import-module "activedirectory" -ea 0)) {
Write-Host "Loading active directory module." -ForegroundColor Yellow
import-module "activedirectory" -ea Stop
}#endif

#users
foreach ($user in search-adaccount -UsersOnly -AccountInactive -TimeSpan 90.00:00:00){
move-adobject -identity $user.DistinguishedName -targetpath "OU=Old Users,DC=some,DC=domain,dc=net" -whatif
}

#computers
foreach ($computer in search-adaccount -Computersonly -AccountInactive -TimeSpan 90.00:00:00){
move-adobject -identity $computer.DistinguishedName -targetpath "OU=Old Computers,DC=some,DC=domain,dc=net" -whatif
}
[/sourcecode]

List Members (and email addresses) of an Active Directory group.

Recently i was asked to list a: all members of an active directory group, and b: pull their primary email address, leaving me with an end report of username and primary email address.

I used dsget to pull the user information from the group, below is the command i used:

dsget group “cn=Groupname,ou=DLs,ou=Exchange Recipients,dc=ie,dc=domain,dc=company,dc=com” -members >> 1.txt

the above command enumerates the “groupname” group in an ou called dls, in an ou called exchange recipients in the domain ie.domain.company.com. if your ou or domain structure is different trim out (or add) what you need.  The -members at the end of the file will dump only the usernames in FQDN format.

Once the script is run check the current directory for a textfile called 1.txt.  This text file will contain the usernames you need in FQDN format like below:

“CN=Tom Thumb (IE),ou=Dublin,dc=ie,dc=domain,dc=company,dc=com”
“CN=Mike Hunt (IE),ou=Dublin,dc=ie,dc=domain,dc=company,dc=com”

In order to get the email address’es i decided not to try and read from the file, instead i just ran the same command again and piped the results to another dsget query.

dsget group “cn=Groupname,ou=DLs,ou=Exchange Recipients,dc=ie,dc=domain,dc=company,dc=com” -members | dsget user -email >> 2.txt

The above will pull the results we saw in 1.txt, but instead it passes it straight into another query (dsget user -email) and sends those results to a text file. 2.txt should contain the users primary email address:

tom.thumb@company.com
mike.hunt@company.ie

Now simply copy the contents on both text files into neighboring columns in excel and you have your report :)

Update: 13/08/2012

An old friend of mine Rob reminded me that this post existed and wondered how to do it with powershell. Luckily This is much, much easier to do with Windows Powershell!

On a server with the active directory module for powershell installed (normally a domain controller), run the following commands: (replace the group name with your own one).

 

[sourcecode language=”PowerShell”]

#######Change the below values#######
$groupname = "My Group Name"
$exportfile = c:tempreport.csv
#####################################

if (!(get-module -ListAvailable | where {$_.name -eq "ActiveDirectory1"} -ea 0)){
write-warning "The ActiveDirectory PowerShell module is Not Installed!"
break}
else{
write-host "Importing Active directory module";import-module activedirectory -ea 0
Get-ADGroupmember $groupname | %{get-aduser $_.samaccountname -properties cn,samaccountname,emailaddress | select cn,samaccountname,emailaddress | export-csv -notypeinformation $exportfile}
}
[/sourcecode]