What is the maximum memory that an IIS6 web site/app pool can use?

Apps under Windows x86 are limited to 2gb/ea. You can change this by adding the /3gb flag to boot.ini, but this can cause unexpected application behavior and should be used with caution. MS officially does not support this (http://technet.microsoft.com/en-us/library/bb124810.aspx)

IIS's memory limits can be set under the Recycling tab in the relevant app pool.

How are you determining that the application is never going over 500MB? If you're using Task Manager, keep in mind that it is rarely (if ever) an accurate representation of memory usage as Windows understands it. Use Process Explorer: http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

There are a number of other relevant questions:

• Are there errors logged in your Event Logs?
• Is it a particular line of code throwing the exceptions? If so, is this OutOfMemory exception actually an IIS problem, or a code problem?
• Are you limiting your application pools at all with the Recyling setting?
• Are there any other services running on this machine?

Most likely, IIS, ASP.NET, and all their associated baggage are running up against the 2GB limit even though Windows Task Manager doesn't show it. Setting Recycling limits or upgrading to x64 will probably make a huge difference.

occasionally get OutOfMemory exceptions from a busy .NET application

The answer to this is too complex to fit in an answer. See "Tuning .NET Application Performance" for a complete treatment of the subject.

Here's a vastly oversimplified (but still quite good) summary from Bruno Jouhier:

Furthermore, the .NET runtime does not let you go up to 2 GB. The garbage collector works by copying live objects, so it needs a fair amount of space to perform its copies.

Edit:

Here's my attempt at an explanation...

If you're wondering what the maximum worker process memory size (as reported by Task Manager) is for an ASP.NET Worker Process on x86, the answer is "it depends".

In any kind of managed code such as Java or .NET, the programmer gives up fine-grained control of memory as a penance for not having to deal with pointers. As a program executes, the Heap and Stack will be periodically cleaned up by the Garbage Collector.

Specifically in regards to ASP.NET, the garbage collector runs inside the same worker process as the website. The GC consumes memory of its own. How much memory is entirely a function of how your application's code is written. One app may be able to use 1.8GB of memory while another may choke at 500MB. In order understand why, you need to profile your specific application.

Any process on a x86 Windows OS is limited to 2gb unless you have set the /3gb switch in your boot.ini file, in which case a process can use 3gb.

From this blog post "Recommendations for SharePoint Application Pool Settings":

Focusing on physical I typically like to limit app pools around 800MB to 1200 MB max on a 32 bit app with very few app pools depending on the number and amount of memory. On a server with 2 GB RAM I'd set it at around 800MB max. On a 4GB of RAM server around 1GB and more if more with a max around 1200. On a 64 bit web front end with 8-16 GB memory I've heard of settings of 2GB of RAM or even allowing it to let it ride, rather than limiting it.

You really need to profile it since these can really grow to process and cache. The greater the amount of memory and the greater the load the higher the worker process will grow. When people ask about configuring the app pool, this is where they are usually asking what the numbers should be. What you are doing here is explicitly limiting the app pool from consuming more memory. Notice this setting is on the recycle tab, there's a reason for that.

When an app pool reaches the max it isn't like the max processor setting. It will cycle the worker process which is like a tiny reboot or similar to an iisreset, but not since sometimes we want this to happen so we can release our memory. You really don't want to cycle more than a couple of times per 24 hour period in an ideal world. I've heard of some trying to cycle right before the morning peak occurs so they have the most amount of memory available, then a cycle right at the end of the day before the backups or crawling begins.

From my experience 800 MB is the treshold for 32bit machines (2-4 GB RAM). It recycles app pools before throwing "out of memory" exceptions.

Make sure you aren't setting the virtual memory size on your app pool. If you set this value to a number outside of the allowable range, it will revert to 512MB. See KB923197.

Also note that if you are running an ASP.Net application, ASP.Net will recycle the pool at 60% of the 2GB memory limit, or 1.2 GB. This isn't your ~500 scenario, but on 32 bit apppools with large memory usage, we sometimes tweek this to get a little more memory.

<system.web> 
  <processModel memoryLimit="80" />
</system.web>