What's The Worst Server Configuration/Setup That You Have Ever 'Inherited' [closed]

Solution 1:

I am yet, in 15 years in the industry, to start a new consulting role at a company to find that they have a "good" infrastructure. That's usually the reason why I'm called in, to put them right.

The usual cause of this mess is non-technical decision makers making technical decisions.

Solution 2:

I did a job a few years ago performing an "assessment" of a small manufacturing company's network infrastructure. During that work, I discovered that their ERP system had never been backed-up. Unbeknownst to them, their former IT contractor configured Backup Exec for daily full backups but never scripted any type of "dump" or stop / start of the database server used by their ERP system, so the database files were always in use and skipped by the backup. As such, for well over 3 years they were performing daily tape backups that had none of their ERP system's data on them. They dutifully changed the tape out, just like the contractor told them, but apparently no one (including the contractor) ever bothered to check to see what was actually on the tapes.

Solution 3:

Once in the olden times, one of our senior admins left our organization and turned over responsibility for the "document imaging system" to me. I was low man on the team, inexperienced, and eager to jump into anything.

It was like the old Coke commercial with Mean Joe Green...I was totally stoked to become the primary (only) admin on a customer-facing Production system and on his way out the door, he was like, "hey kid, catch" expect he tossed me a wad of crumpled papers with some logins and a telephone number for support instead of a sweaty towel.

The euphoria quickly wore off...the system was comprised of 2 servers running a database, a share, about 6 workstations with scanners and processing applications, and a webserver and app users logged in to for referencing documents. It was an unholy mishmash of apache and java and at least two types of scripts running on Windows SQL Server. Oh yeah. We'd also paid for a series of "customizations" that often broke down and that their support folks were always blissfully unaware of.

Short list of The Good Times:

• The app had memory leaks and would hang.
• It was integrated with our ERP through a series of feed-files pumped back and forth over nightly FTP jobs. The sequence of the feed-generations, processing, file-push-pull, and database updates on both ends depended on careful timing between some app scheduling, SQL Server jobs, and nightly crons on the remote ERP system. If updates in either direction failed, whole departments were at a stand-still b/c their "reports" didn't get spit out of the printer or, worse, contained inaccurate info that would result in customer complaints.
• The SQL Server had no maintenance jobs configured and log truncations were manual.
• Sometimes the license file for the app randomly "expired" and locked everybody out.
• Sometimes the internal user roles got "confused" and folks would log in and see (and be able to use) the admin interface buttons. (those calls were great..."Dan...I see some new buttons...Should I click them?")

Little if anything was documented and I discovered each wrinkle when something broke. Like say...the reports were wrong or didn't print. Or Desktop pushed a new version of the JVM and nobody could scan. Or somebody kicked the dongle off the scanning workstation and the app crashed. Or the log file system got full. Or data from an OCR extraction crashed an app due to incorrectly capturing something and submitting it as something illegal. Or finding out that there were about 3 dozen tickets open with support for various departments and many had been open for months. Etc etc. I discovered new, important things at the rate of 4-5 a week and began to very quickly learn the ins and outs of that app and its needs as well as enough SQL Server to keep the db moderately healthy.

The best part was when I was invited to the internal User Group meeting to "welcome" me to my new role. I kid you not. 30 angry users in a circle and I got to sit in the middle.

It was a rough but I learned quite a bit very quickly. All the pain aside, it was a great opportunity. Part of me wishes it hadn't been so trial-by-fire but maybe I wouldn't have learned so fast.

Sorry that was so long...but ahh...it's like therapy ;)