IDS for Windows Server 2008?
I am sure my Windows Server 2008 box is constantly under attack both at the network level and web application level.
Question is how do I detect these attacks? Is there any light-weight software available which can monitor the server?
Note: I am running this on a VPS so the monitor program will have to run on the same server.
IMO, you can never fully trust software on the server being monitored for this type of information. You need a transparent layer in between all traffic and your server. You could also do "sideband" monitoring by mirroring all traffic that hits your server and analyzing that. That really doesn't offer much potential for protection though.
I suppose you could look into installing Microsoft IAG or TMG on the server, but that's a bit heavy. TMG would address the network level attacks and IAG would specifically address the web app layer.
I would suggest a multi-faceted approach, utilising Snort as a NIDS (network intrustion detection system, and something else as a HIDS (host intrusion detection system).
On the HIDS side of things, Pixelicious has already mentioned Tripwire, but as alluded to, it's a very expensive solution.
OSSEC is a powerful Open Source HIDS that may well fit your needs, and commercial support is available if required (through Trend Micro).
Ionx Verisys is a commercial, but lighter-weight and less expensive alternative to Tripwire.
I'm going to assume that you won't have to resources to use any enterprise grade IDS like tripwire? which would definitely be effective. What I can suggest is that you log all the requests you receive, so in IIS be sure to log and review your http requests. Can you limit the IP ranges you're expecting connections from?
Can you authenticate your web app? if you disable anonymous access you should be able to greatly reduce the attacks on your app. If not, look at lowering the privileges of your IIS service, and the app pools.
If you're seeing probes on your network, identify network services (netstat -an will help) switch everything you can off. Switch on your windows firewall; look over the rules, limit it to the bare minimum.
There's no application you can install or device you can buy that will just switch on and make everything ok. Your best defense is to actively review your system.