Too much memory getting used
I suggest downloading the Sysinternals Suite from Microsoft. It includes several utilities that will give you more information than Task Manager does. Take a look at the Process Explorer, RAMMap, VMMap, and Process Monitor utilities. To get the most information from them, you may need to do some configuration to change what info is displayed.
Also, since you're interested in Windows memory, here is a series of blog posts from Mark Russinovich, one of the Sysinternals developer/founder. It will help make sense of the info the utilities provide.
Windows 7 uses 'unused' memory to precache things to make things run faster. Should you need this memory, it'll get freed up.
Its a feature, not a bug ;)
From technet
After you’ve used a Windows Vista system a while, you’ll see a low number for the Free Physical Memory counter on Task Manager’s Performance page. That’s because SuperFetch and standard Windows caching make use of all available physical memory to cache disk data. For example, when you first boot, if you immediately run Task Manager you should notice the Free Memory value decreasing as Cached Memory number rises. Or, if you run a memory-hungry program and then exit it (any of the freeware “RAM optimizers” that allocate large amounts of memory and then release the memory will work), or just copy a very large file, the Free number will rise and the Physical Memory Usage graph will drop as the system reclaims the deallocated memory. Over time, however, SuperFetch repopulates the cache with the data that was forced out of memory, so the Cached number will rise and the Free number will decline.
The numbers you see are totally imprecise and should only be taken as indications.
An operating system is an extremely complex entity and a simple-minded program like Task Manager can never total the entire usage, many parts of it are simply inaccessible in User mode under which it runs. Some of the elements that are inaccessible to it, or are simply impossible to attribute to any "task", are the hardware segmentation tables, shared-memory allocations, kernel-allocated memory, page-memory allocations, device-memory and many more.
You will find more explanations in How do I interpret the Performance tab of Task Manager?, but I choose to quote this part :
What do you expect from a program calling itself the "Task Manager" anyway? There is nothing called a "task" in the operating system kernel - the OS has "processes" and "threads". DOS had "tasks". The Intel hardware has "task" structures, but the OS doesn't use them because it's faster for it to do it itself.
The Task Manager is a very old Windows program, has problems even attributing functions to services, and the numbers it shows should be taken with a large pinch of salt. It is probably missing out on large areas of memory that it cannot manage to count or attribute to any "task".