Centos went read only suddenly, on vmware....scary

I have a virtual Centos 5.6 on VmWare. SUDDENLY, it stopped working and it seems the file systems went read only.

cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 ro,data=ordered 0 0
...

It seems here that's a known "problem". But I tried the "solution" and I got:

mount -o remount /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only

I don't know exactly what to look for in dmesg...here the only reference to "read-only":

grep -B 10 "read-only" dmesg

Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
Using IPI No-Shortcut mode
ACPI: (supports S0 S1 S4 S5<6>Time: tsc clocksource has been installed.)
Initalizing network drop monitor service
Freeing unused kernel memory: 228k freed
Write protecting the kernel read-only data: 413k

Of course everything is stopped, web server db, not working etc. Worst problem I ever saw, completely in the dark.

Any suggestion welcome.

Now I tried

fsck /   

too many errors asking confirmation

fsck -a / 

ended with

/dev/VolGroup00/LogVol00: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)

and then

fsck -y /
/dev/VolGroup00/LogVol00: ***** FILE SYSTEM WAS MODIFIED *****
/dev/VolGroup00/LogVol00: ***** REBOOT LINUX *****
/dev/VolGroup00/LogVol00: 76279/11250176 files (3.6% non-contiguous), 1638779/11247616 blocks

hmmm now...it doesnt reboot??? shutdown -r now does nothing....reboot is even better:

reboot: error while loading shared libraries: libxml2.so.2: cannot open shared object  file: No such file or directory

This is because ext3 on older revisions of EL5 is susceptible to filesystem corruption. Update your kernel/OS at at least EL5.8 to get past the issue.

I didn't notice this until I worked in a hosting environment with a large number of unpatched RHEL5 servers. We had weekly occurrences of this.