How do you recognize a good system administrator?

In the company I'm working for, we need to get system administrators. However, we are a programming development company and it turns out that we have no idea how to tell a good system administrator from a bad one*. We just needed someone to set up the server, plan the layout of the LAN cables and to set up policies on the security of the Wi-Fi.

We didn't realize that we have a problem with our hiring until we found out that the two administrators we hired didn't do the job properly. We found out we have problems two months later when:

• we started getting static on the phone and we traced it to the cabling.
• a visitor told us that the network security is ineffective and demonstrated this.
• we have to replace the server they recommended since the old one was inefficient for our company.

Is there any standard way of recognizing a good system administrator?

Are there any interview tests that we can give to weed out the poorly skilled ones?

* You would think computer programmers would tell the good technical staff from the bad ones but programming and system administration are two different fields.

Solution 1:

Here are some ways to recognise a good system administrator.

• They are able to talk about previous systems they have administered in a way that makes sense to you, a technically-capable non-sysadmin. A good sysadmin needs to be able to communicate with other network users and see the big picture at the same time as being fully aware of all the details. If they can't explain in a structured and clear way what they did and why in a previous job, then they won't be able to explain to you their decision-making rationale when working for you. Basically, they should be able to talk all day about a specific system without ever losing their audience.

• They are obsessed with avoiding single points of failure. At any point when they are describing a system they administer(ed), stop them and ask "What could have gone wrong with this part of the system and how did you mitigate that risk?" Their answer should be detailed and show that they had thought it through carefully already. They should also be enthusiastic about answering that question, because good sysadmins love thinking about ways to avoid catastrophic failure.

• They have a healthy scepticism of the new, the cool and the untested. They are also hugely keen to trial new solutions and are always doing so. However, their standard toolbox is staid, safe and involves plenty of testing.

• They can remember times their systems have failed and answer five whys without having to think. Every sysadmin has made mistakes that led to downtime; the good ones have thought about both technical and systemic reasons why it happened.

• They document their systems with the same level of obsessiveness that a teenage diary writer documents her crushes. If possible, ask to look at their documentation for previous systems they have administered.

• I've no idea how to test this at interview, but they are calm in a crisis. Perhaps you could wait till they visit the lavatory, then jam the door and set off the fire alarm.

Solution 2:

It's hard for programmers to recognize good programmers; e.g. it takes companies like Microsoft and Google a day's worth of interviews to satisfy themselves, even after initial screening. Similarly, it's hard for system administrators to recognize good system administrators.

On that basis, I claim that it's extremely hard for programmers to recognize good system administrators.

That doesn't mean you should give up, however. But why not enlist the help of a system administrator to help you in the interviews?

It may sound like this creates a circular dependency (because how do you know who to ask to help you?), but I think it should be relatively easy to find a friendly sysadmin from among your professional contacts who might be willing to help out with something like this. You probably wouldn't want them to make the choice for you, but they should be able to give an informed assessment of their technical and other relevant abilities.

Solution 3:

Characteristics that I look for in a System Administrator, although a bit *nix in nature

• Obsessively organized
  • Lists of things to do
  • P-Touch labels everything
  • Everything has it's place and it upset if said thing is out of place
  • Very, very punctual
• Minimal coding abilities
  • Can whip up a simple bash script (variables, if/then/else)
  • Can modify a complex bash script
  • Can compile httpd from source
• Certifications
  • They mean more for System Administrators (than for us coders)
• Attitudes towards programmers
  • Cautious, if not downright suspicious
  • Intrigued by new requirements, willing to negotiate
  • Will not give coders root, no matter what
• Has strong opions about monitoring systems
  • Prefers something SNMP-based. (And it is not "What's Up? Gold")
  • Needs access from home and mobile phone/PDA/netbook/etc so the system can be monitored while on holiday
• Comfortable with CLI, if not preferred over a GUI

Solution 4:

I consider myself a good system administrator, and one of the reasons I am now is because I have made all the mistakes you've listed. I won't ever make them again because I learned from every one of them.

If you want a good system administrator, find one that has already made all the mistakes and learned from them. If they don't like learning from their mistakes, they aren't good sysadmins. If they say they've never made a mistake, they are either lying or not a good sysadmin.