How can I be notified if somone is attempting to hack into a Linux server? [closed]

If you're worried about attacks, simple monitoring isn't enough. Imagine an attack comes in at 2am, or whenever you're off the clock and asleep. How many password guesses can be made before you check your email?

Too damn many. fail2ban and other programs will automate the policy you're after. Logwatch can look for unusual behavior, but it's mainly intended for discovering logged errors.

I do log monitoring with logwatch to look for suspicious login activity. I have the system locked down pretty tight so I mostly "catch" our security office doing their regular scans. There's also an open source version of TripWire that would be useful for monitoring changes to selected files, though this is only useful after they've broken in to let you know what's been compromised.

I think you're looking for snort

It's an intrusion detection system or ids

Takes a bit of configuring and then tuning to minimize false alerts (or to address problems), but there are many tools available to help with that. There are also websites where you can subscribe to new sets of "rules", in order to keep on top of the latest hacks.

Along with that you might want to use snort log analyzers such as base or acid. I think there's also an all-in-one gui solution called sguil as well, that you might want to look at.