Best system administrator accident [closed]
I had fun discovering the difference between the linux "killall" command (kills all processes matching the specified name, useful for stopping zombies) and the solaris "killall" command (kills all processes and halts the system, useful for stopping the production server in the middle of peak hours and getting all your co-workers to laugh at you for a week).
I was in charge of our corporate web proxy which at the time was Netscape's product. While playing around in the admin forms (it was a web based interface) there was a big (and I swear it was red) button that said Delete User Database. No problem, I thought. Let's see what the options it gives me are when I hit that. Surely there will be a confirmation prompt if there are no options.
Yeah, no confirmation. No options. No more users.
So, went over to Mr. Solaris Sysadmin and said that I was in desperate need of a restore from tape to which he replied, "I don't back that box up."
"Uh, come again," I retorted.
"I don't back that box up. It's on my list of things to add to the backup rotation but I haven't gotten around to it yet."
"This server's been in production for nearly 8 months!" I screamed.
shrug, he replied. "Sorry."
Many years ago the company I worked for had a client which ran a nightly backup of their NT 4.0 Server to a Jaz drive (like a high capacity zip disk).
We set up a batch file, which ran as a scheduled job overnight. Every morning they'd collect last nights disk from the drive, and before they left in the evening they'd insert the next disk in the sequence.
Anyway, the batch file looked something like this (the Jaz drive was drive F:)...
@echo off
F:
deltree /y *.*
xcopy <important files> F:
Anyway, one night they forgot to put the disk in. The change to drive F: failed (no disk in drive), and the batch file continued to run. The default working directory for the batch file? C:. First time I've ever seen a backup routine destroy the server it was backing up.
I learned a little something about sysadminning (and exception handling) that day.
Jim.
PS: The fix? "deltree /y F:\*.*".