IIS7 ASP.NET application - 2 identical apps in 2 identical app pools, 1 is responsive and 1 is not

I have an ASP.NET (v4.0) web app that is installed in a virtual directory (as an application) and is hosted in it's own app pool. This is repeated for each instance of the app (i.e. per customer).

The app pools are integrated (not classic) mode and LoadUserProfile is set to true. Otherwise, default settings.

Each instance currently has it's own copy of the code/config, and it's own data folder (basic file read/writes).

1 instance of this app runs well (operation used for comparison takes ~4 seconds). Every other instance runs slowly (from 10-25 seconds for the same operation).

If I move the slower instance to the "fastest" app pool that instance springs to life. If I move the faster instance into the slower app pool that instance slows to a crawl.

The app pools were created in the same way initially - manually. I later used the powershell copy routine to ensure an exact copy of the faster app pool and still the same behaviour. Comparing the apppool.config files shows they are identical barring the virtual directory assignments.

There are no shared resources that are being blocked, so far as I can tell, and I tested that by shutting down the performant app pool and restarting... slow is still slow, and then when I restart that app pool (so it's loaded last) it's still faster...

To further isolate the problem, I would suggest running Wireshark (or other packet analyzer of choice) on the host system, for two sessions. Assumption I'm making is that each app pool has either a unique IP assigned to it, or a unique port.

First get your baseline performance by filtering on the IP:port of your fast app pool. See what the traffic to and from the app looks like under "normal" conditions.

The second run, you will need to capture traffic from the slow/unresponsive app pool. If all network routing and such is correct up to the box, you should see repeated requests in one direction, most likely to the app from elsewhere, BUT if your app is something that makes a lot of requests to another server, your traffic may be heavy on egress instead of ingress.

This test will tell you if the problem is within the app, or if it's TCP/IP related issues that result in requests to the app timing out due to low/no communication.

Correlate the timestamps of your tests with the server's event logs and (if applicable) tracesink logs, and you should be able to zero in on the problem.