32-bit SQL eating up RAM on 64-bit OS until SQL is brought to its knees

This configuration setting should be a good start:

Edit: Also, keep in mind that limiting the amount of RAM that SQL is allowed to use will save memory for other applications, other instances, and the OS itself... but by limiting the amount of RAM SQL is allowed to use, you may just hit the performance wall more quickly. SQL will start paging data out to disk when it runs out of RAM, regardless of if you are putting a hard limit on the amount of RAM SQL is allowed to use or not. If you want performance, you want to avoid having to use the computer's disks. This means adding more RAM, using 64-bit instances so they can take advantage of the extra RAM, and having more efficient indices and queries. You already said you can't do the first two - the next thing would be to find a DBA to try to optimize your data and queries so that they don't use as much memory.