ASP.NET performance decrease between 32bit/64bit server
As per documentation MS reommends running 32 bit app pools on 64 bit server as a web app should not use more memory anyway (internally).
32 bit apps are always more performant - smaller pointers mean less memory needed for a specific set of instructions which means better cache hit ratios.
64 bit processes only make limited sense - when a process really needs more memory. if you keep your web front end doing just that (front end) there is no reason it ever needs that much memory ;)
On top, it makes some sense to actually upgrade to more performant versions. IIS6 sucked big tiem performancewise - IIS 7.5 is a LOT better, ESPECIALLY if you run ASP.NET / managed code.
We tried this on our Windows 2K3 server and got the same thing, I was not here at the time but it something to do with they way MS implemented 64Bit .Net in 2K3(not sure what but the person that told me is a MVP so I will take his word on it.)
And just to let you know that if you move to Win2K8 you might see the same thing as I did that with our servers here and got the same performance issue. From what I have been told and a bit of reading to take advantage of 64Bit .Net the use of integrated pipelines is recommended (http://msdn.microsoft.com/en-us/magazine/cc135973.aspx)
Sorry this does not answer your question but I thought i would let you know that you are not the only one seeing this issue.