User reduced LVM logical volume without resizing filesystem
I received an email yesterday that one of our users was trying to make room for a heartbeat/clustering package which requires its own partition to act as a voting disk. To do this, he attempted to reduce the size of the root partition's logical volume, and then create a new logical volume for this purpose. However, he forgot to resize the filesystem first (or include the -r switch in the command). He also forgot to unmount the root partition by running this process from a rescue cd. The system is now refusing to boot into the OS with the following error:
Either the superblock or the partition table is likely to be corrupt! Unexpected Inconsistency; run fsck manually.
The system them drops the user into single user mode.
Is it possible to rescue the filesystem, or is it hosed? Its running ext3.
If a filesystem has already been initialised on the new LV then you're probably hosed, because data will have been overwritten. You can still try to recover it, but I wouldn't be too hopeful. If a filesystem hasn't been initialised then it's theoretically possible to recover it (but I haven't tried this myself).
Firstly, make a backup of the entire drive, you'll need it in case the recovery goes wrong. The next step is to try find the LVM metadata backups which LVM creates in /etc/lvm/archive before making changes. If the root volume isn't accessible you can try running e2fsck to get it mountable, and hope that the backup files were stored near the beginning of the volume. If you do this you'll want to continue recovery from before you ran e2fsck (i.e. restore from the backup after getting the backup files).
If you managed to get a backup file, restore the LVM configuration with vgcfgrestore. If you couldn't get a backup file you're just going to have to hope that the initial volume was completely sequential. Remove the new LV and then extend the old LV to its original size.
Once you've got the old LV back to the correct size, cross your fingers and run e2fsck. And you really need to make that backup first, you're probably not going to get this right on the first attempt.
What I would try:
- Use a rescue system, eg. GRML or an Ubuntu live system.
- Resize the volume back to it's original size and hope that LVM assigns the same physical disk space. This should work if no other modifications have been done to the volume group in the mean time.
- Make an image, or at the very least, a snapshot of the volume.
- Try to fix it with
fsck. - Hope for the best
In all likelihood, I would assume the file system to be dead but this might help with a awful lot of luck.
Today I was that user... Fortunately it was on a testing environment, so I kept a clear thought process ;)
I was able to resolve the error by reversing what I had done. I reduced my home lvm by 100G, did not resize, then expanded my root lvm by 100G and still did not resize the FS.
I booted up from a liveCD (CentOS 6.4), shrank the boot by 100G, resized, expanded the home by 100G, resized. Rebooted, and everything was back to normal.