Detecting server abuse
I think you're looking for an Intrusion Detection System (IDS), or maybe an Intrusion Prevention System (IPS). Have you looked at Snort?
CSF/LFD (http://www.configserver.com/cp/csf.html) are fantastic (and free!) and work great with cPanel boxes. It also works just fine on non-cPanel boxes.
Takes less than 5 minutes to setup on your existing box. Tweak the defaults for your environment and it'll automatically block IPs and/or kill processes that go beyond the bounds you set, and send you an email of what it did.