Minimal Windows 2008 or 2012 memory usage
Solution 1:
I just did a clean installed of Web Server 2008R2 Core in a virtual machine. It is currently using 253MB of RAM exactly. I do not curently have access to a Server 2012 .iso to test.
The thing is with Server Core is that there's is no GUI which cuts down on RAM (and attack surface) but you have to get quite adept with managing it. As explorer does not run it might make your web server software (whatever you chose to use) difficult or impossible to use. Using IIS or any other web server would make the RAM increase significatly - I would say 256MB RAM for a miniumal install, and 512MB for a "useable" server - more if you were running some sort of database.
Of course, you'd likely be better off with using some Linux variant such as CentOS for running a web server. When I got my first VPS I went for Windows over Linux as I was more comfortable with using it and the cost of me paying for the windows license was made up by the fact I didn't spend any time fumbling around with CentOS and risking a compramised server.
I just went back to the VM after typing that and it has gone up to 272MB.
Solution 2:
In my experience both OSes are fairly low usage, for a Windows box. An even smaller foot print install would be to use the core installation mode to remove the need for an actual Windows UI layer as well, however that may not play nice with your 3rd Party web server.
In all reality though, if you plan to use a non-IIS web server there really aren't any reasons not to use a *NIX OS and run the web server on those boxes. Linux virtual machines are often cheaper than the Windows counterparts simply because the cost per hour includes the Windows licensing.
Regardless of all of this, you would simply need to understand the minimum amount of RAM allowed by each OS and then determine the minimums for your web servers and other applications. Of course if you use the minimums you can expect to see low to horrible performance across the board.
Solution 3:
I just took a VM with 2012 Server Core, SQL-2008 R2 and several ASP.NET sites on IIS and limited the RAM to 256MB, it still all works but it is noticeably slower.
I would set up a test VM with your applications and then run some performance and scaling tests against different amounts of RAM.
I would recommend the core option as well, even though I don't think the memory footprint is that much lower. Even on a Full-GUI server, Windows Explorer is usually not loaded. But when you log in Explorer needs more memory than cmd.exe.
Server 2012 core has a bigger storage footprint than 2008 R2 core, but I think the memory footprint is about the same. I have no hard numbers for this though.