Remote Fedora system unresponsive, odd but consistent behavior when trying to log in
This sounds like your system is under very heavy load. It's most likely swapping a lot or otherwise using disk I/O.
What I suspect that happens is that for some reason*) your site suddenly receives a flood of http requests to some heavy dynamic page, which makes Apache consume all of your server resources. The real reason could very well also be a heavily loaded and slowly responsing database server, which would ramp up the page load times at your web server, which would make Apache consume more processes, which would lead to server overload, which would lead to situation you are observing.
Are your sites usually having a lot of traffic? A ballpark figure would be nice -- are we talking about thousands of hits a day, tens of thousands hits a day, millions...?
Do you have any resource graphs of your server? Something like snmpd+mrtg or Cacti graphing CPU, memory, disk usage? If, are there any spikes? Or if you have sysstat
package installed, then you might have sar
snapshots available -- try command sar
. It would save various information about your system every 10 minutes or so and through it reports you might see what was going on.
And is this Fedora installation a separate physical server or is it somehow depending on the VMware environment you are talking about? Could it be that some part of your VMware infrastructure gets overloaded every now and then and THAT is the real reason for your problems?
Also, after the dust settles, I would investigate the logs to see if there were any signs of kernel yelling about OOM (Out of Memory) Killer, or if httpd logs would look suspicious.
Usually if I'm unable to see anything suspicious in httpd logs by my own eye, I will let Webalizer or similar analyzer to run through that log and see if any URL's are popping in the output. Or I just pipe the log to apachetop
and see what was happening.
*) Reason can be a DoS attack, a badly behaving PHP script, an erroneous .htaccess file making your ErrorDocument page to recursively call itself, Slashdot/reddit effect, or something completely else.