We had this problem recently with our XenApp servers, as part of the latest service pack DEP (or Data Execution Protection) has now been enabled for all applications accross all platforms and this can cause havok in a terminal services environment.

The problem with Dep is simple, its not a Reg Key, its not an environment variable its a system setting loaded on startup from the boot.ini file. To make matters worse its a kinda complicated setting as it sounds opposite to what it is exactly doing.

Take this for example, this is a server with DEP enabled:

multi(0)disk(0)rdisk(0)partition(1)WINDOWS=”Windows Server 2003, Standard” /NoExecute=OptOut /fastdetect

What confuses me with the above statement is why Opting out means DEP is enabled? To disable DEP you need to change this switch to OptIn, which makes no sense to me at all.

To automate this change, i used a tool called RPL.exe, this is a direct translation of the Unix command/tool to replace text inside of a text file. Using RPL i could search the boot.ini for the optout statement and replace it with the optin (aka turn off Dep).

Because boot.ini is a system and read only file by default, i needed to use the attrib command so first i set about removing the read only, hidden and system file attributes on the file in order to edit it:

attrib -r -h -s c:boot.ini

once the file was editable, i now performed the rpl command to search and replace:

rpl.exe -i /noexecute=optout /NoExecute=OptIn c:boot.ini

Once the file had been changed if neccessary it was time to set the file back to read only, hidden and system before closing the script:

attrib +r +h +s c:boot.ini

And voila, next reboot and Dep is gone