How to limit the memory used by an application in IIS?

IIS will respect both of those limits.

If you set a 500MB private byte limit, as soon as a worker process attempts to commit 501MB, IIS will spin up a new worker process and kill the old one.

If you set a 3GB virtual memory limit, as soon as a worker process attempts to reserve 3.001GB, IIS will spin up a new worker process and kill the old one.

If you are on a 64bit platform, you should be aware that ASP.NET applications aggressively reserve virtual memory. As an example, I have an app on a farm that uses only 88MB of private bytes, but its sitting at 5.4GB Virtual Size right now. I believe the virtual memory reservation is a function of physical RAM on the server. It's also important to understand that on a 64bit platform, reserving large portions of virtual memory has zero performance impact.

Basically, if you are having memory consumption issues on an IIS server, the setting you want to limit is Private Memory/Bytes, this is what corresponds to actual memory usage.