I accidently zipped my whole server

Alright if anyone wants to play god and work miracles i'm down.

So, i was given the task of making a script that found files that are older than 6 months, zipped them and then deleted them. On my way in making tihs script i ran this :

find / -type f -mtime -400 ! -mtime -180 | xargs gzip blablabla

And that gave EVERY SINGLE FILE a .gz extension. Now i undid it as soon as i noticed but it was kinda too late. Upon completion of the command none of my bash commands would work because the $PATH variable emptied itself. I tried lots of things before realising what was the problem.

So ater unzipping everything i'm still unable to boot. I've managed to make it to grub rescue, after wich i followed online instructions for :

root (hd0,0)
setup (hd0)
kernel (hd0,0)/boot/vml[...]
initrd (hd0,0)/boot/initrd.im[...]

After wich my linux partially boots but gives me the following errors :

Begin : Running /scripts/init-bottom ... mount : mounting /dev on /root/dev failed : No such file or directory
mount: mounting /sys/ on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed : No such file or directory
Target filesystem doesn't have requrested /sbin/init.
No init found. Try passing init= bootarg.

I've tried repairing the filesystem, i've booted from 3 different LiveCDs/ Rescue disks, i ran boot-repair from 2 different dicsc. I did force fscks...

I'm really out of ideas and i NEED to get this server to at least boot so i can recover my SQL databases. I'm desperate for help, i'll even pay if need be.

I've been lurking forums for 3 days straight all day to find a possible solution and i'm still at the same point... Help please?

Solution 1:

This will depend on whether the filesystems are repaired enough for you to be able to mount those partitions from a LiveCD. Don't bother trying to boot the system yet. First, mount the partitions and unzip all the .gz files. This will give you working copies of init and system binaries. Then you can use grub to repair the boot sector. Then boot to single user mode and fsck the filesystem again. If that works, you'll have a running system. You'll also have a bunch of unzipped files (like man pages) that really should be zipped, but it's better than having an unbootable system.

If you can't mount the partitions from a LiveCD, then you are unfortunately out of luck. Nothing will recover your system at that point.

Solution 2:

The first thing I'd try is running a LiveCD environment and just attempt to ungzip everything, hoping that would return the system to a bootable state. Note: I'd be concerned about potential data corruption if the original gzip process was interrupted.

Otherwise I'd try to migrate the DB to a new system as others have suggested but as you've encountered there may be labor intensive dependency and configuration issues that will need to be resolved individually.

Solution 3:

The general consensus here, that you should just mount the disk in a working system and rescue your files, is not wrong. It's the sensible thing to do. But the other way is more fun, and very educational. I've learned a lot while fighting my way out of messy situations where other people would have just given up and reinstalled from scratch. (Not on a server that other people are depending on though...)

Anyway, so far you have an initramfs(initrd) that runs. That's a good start. But it can't complete the handoff to init because init is now init.gz maybe? To make any progress, it would help to know exactly what Linux distribution you have, so we can look up what tools are available in its initramfs for emergency use.

The error messages you presented look like they could have come from Debian's initramfs. If it is Debian, then you should have gotten an (initramfs) shell prompt on the next line after the last error. If you did, you should look into what's going on with those failed mounts. is /root/dev missing? (/root is where your normal root fs should be mounted during the running of the initramfs)

If you didn't get the shell prompt, then what came after No init found. Try passing init= bootarg. will be interesting. Even if it was nothing but a blinking cursor, that's a clue. If it seems totally frozen, try to get some information on what processes are still around using magic sysrq or Ctrl+ScrollLock.

The Debian initramfs also lets you request a shell at a few special landmarks by adding a break= parameter to the kernel command line. For example, to get a shell before the Running /scripts/init-bottom line, use break=bottom.

Aside: I don't know how the find command could have gzipped every file... it looks correct to me for the purpose of selecting files between 180 and 400 days old.