Weird memory usage in Windows Server 2008 R2

Is this a 64bit server - do you have the lock pages in memory local policy enabled? SQL is likely consuming the rest of your memory If you look at the perfmon counters you will see the memory allocation

Here is an article that explains it in depth

You can also view the counters in SQL

SELECT
    object_name
   ,Counter_name
   ,cntr_value
   ,ROUND(( cntr_value * 8192.0 ) / 1048576, 0) AS cntr_value_MB
FROM
    sys.dm_os_performance_counters
WHERE
    object_Name LIKE '%Buffer Manager%'
    AND RTRIM(counter_name) IN ( 'Free pages', 'Total pages',
                                 'Database pages' ) 
UNION SELECT
    object_name
   ,Counter_name
   ,cntr_value
   ,ROUND(( cntr_value / 1024 ), 0) AS cntr_value_MB
FROM
    sys.dm_os_performance_counters
WHERE
    counter_name IN ( 'Target Server Memory (KB)',
                      'Total Server Memory (KB)' )

Quick test: restart SQL Server.
Another quick test: restart IIS.

You'll know for sure if one of them is the culprit, or if you have to look somewhere else.

It may help to use RamMap to see where your memory goes to.

Are you actually experiencing any memory-related problem?

Does memory usage ever become higher than the actual installed memory, or does it just fill up and stay there?

If memory just fills up but you're not experiencing any problem, then it's just cache doing its job; memory will be automatically realeased if and when you'll actually need it.

If memory usage goes above the physical limit and the server grinds to a halt, then you're experiencing some memory leak and should definitely debug it.

Some interesting articles on the "cache" topic:

http://blogs.msdn.com/ntdebugging/archive/2007/11/27/too-much-cache.aspx
http://blogs.msdn.com/ntdebugging/archive/2007/10/10/the-memory-shell-game.aspx
http://blogs.msdn.com/ntdebugging/archive/2009/02/06/microsoft-windows-dynamic-cache-service.aspx