IIS 7.x Application Pool Best Practices
1) It seems advisable to have an application pool per website. Are there any caveats to this approach? Can one application pool, for example, hog all the CPU, Memory, Etc...?
This is a pretty good approach; there aren't any good reasons that I can think of to have different "sites" (applications) share the same pool. Unless they need to share a single resource of some sort. One application could theoretically hog a lot of CPU or memory, but changing how the applications are pooled won't really affect this much.
2) When should you allow multiple worker processes in an application pool. When should you not?
This is best left alone, using the default settings. Unless you really know what you're doing this can actually negatively impact your website/application.
3) Can private memory limit be used to prevent one application pool from interfering with another? Will setting it too low cause valid requests to recycle the application pool without getting a valid response?
a) Theoretically
b) Yes setting it to lower can have negative effects. Again, unless you have specific needs, and know what you are doing, just leave these alone.
4) What is the difference between private and virtual memory limits?
That's very complicated, here's a quick post I found that might help: http://cybernetnews.com/cybernotes-windows-memory-usage-explained/
5) Are there compelling reasons NOT to run one application pool per site?
Again, only reason I can think of is if there is some sort of "shared resource" the multiple applications need, then you would want to run them in the same process.
For general purpose applications and websites, IIS is pretty well set up with it's default values.
****UPDATE****
In regards to your request for additional information on #2, you shouldn't do this unless you have a specific need to do so. Even with server actions that take a long time, requests are served up using multiple threads, and you would want to use "Async Requests" to handle long running tasks (which frees up a thread pool thread to handle other requests). Realistically, I can't think of any good reason to allow multiple processes for the a single pool.
Once you're start talking multiple processes, then you potentially run into things like: losing session state because a session is alive in process 1, but the request is being handled by process 2. Or even worse, you have to figure out how to do some inter-process communication, which is a real pain.
No matter what you come with in regards to a reason for multiple processes, I would be willing to bet there is a better way to deal with it (rather than firing up another process).
I always configure a dedicated application pool for a web site. Low-cost web site hosting scenarios are where having a large number of sites per application pool makes sense.
The memory limits are really only primitive safety thresholds to prevent a site from consuming all system resources. Note that this is more of a potential issue on Windows 2008 R2 x64 than on IIS 6.0 x86, because x86 applications had a natural 2 GB memory ceiling. It is much easier on IIS 7.5 for an application with a memory leak to consume vast amounts of memory.
I'm also not a big fan of recycling application pools. If I have an application pool and I'm the only application running, if there is nothing wrong with our code, there is probably no need to recycle the app pool. And if there is a defect in the application, the ultimate appropriate action would be to fix the code.