Windows 2008 R2 Server using 100% RAM, can't find the problematic process

OK so I have a W2008 R2 server running a single website with about 100-200 visitors daily, and it also has MSSQL 2008 running on it for the same website.

Recently, when the server starts it uses a normal amount of RAM (about 40-50%) and then slowly but surely it climbs until it reaches 100% and no one can access the site anymore. Even restarting it takes about 10 minutes just to accomplish.

Looking in the task manager, I see no specific culprit that I can pinpoint and say is causing said memory leak.

The weird thing is, this website has been running since 2007 or so, and it had times when the site had between 2000-4000 unique visitors per day with over 1.2M pageviews and it had no problems what so ever.

Now if it runs 2-3 days without restarting it's a miracle.

Any help would be greatly appreciated.

EDIT:

OK so I limited my SQL server to 1,400 MB of RAM, and I'll use resmon.exe once the problem happens again.

However, I'm more of a programming-side guy and I'm quite a newbie when it comes to server management; what exactly should I monitor with perfmon?

10 bucks says its the SQL server. Try settings it's max memory to below the system limit. Sometimes this is needed on shared systems.

Normally you want SQL to consume everything available, but this is one of the situations where it might not be for the best.

You might be able to get a slightly better overview of your RAM usage (and a possible process using it up) by having a look at the windows tool resmon.exe.

Also you could track the usage over a few days via perfmon.exe, which may help determining when exactly the RAM usage increases. Perhaps that coincides with some other program task or server event.