How to recruit a linux guru

A Beginner:

• Has less than 4 years experience.
• Has to rely on binary packages for everything
• Has never seen an old kernel (i.e. only knows 2.6.x series)
• Hasn't figured out that the commands and directory locations are different in each distro; often, they only know of one they are starting out on, and can become confused when their environment has switched.
• Can't script common commands and often do everything manually.
• Needs assistance in performing diagnostics on a troubled system, although they function independently on lighter issues.
• Is still learning from others things that the "Seasoned" admins already know.
• Has a demeanor that is still "green' - they are self-assured (rightly so), but appear cocky to some. This can lead to friction with end-users, developers, and management. Troublesome end-users can often get them to do something that a seasoned admin would immediately deny. Developers don't have much to talk about with them, but may teach them a thing or two about scripting. Management usually wants someone more seasoned and will not bother them unless there are limited choices.
• They often do not have a complete picture of your core business and how it generates revenue, although they do understand procedural-level positions in the company. As such, they can identify the needs of regular staff throughout the company, but do not necessarily understand the interactions of all company units.

These are the admins that start out in junior level positions.

A (stereotypical) impression: "This person's got potential, they just need time to make it shine."

A Seasoned Admin:

• Has 5+ years experience.
• Can download and compile tools/utilities/services, and can recompile a kernel
• Has seen older kernels (2.2 and 2.4 series)
• Can adapt to a different distro, or has experience in 2 or more distros.
• Can do simple scripting to automate tasks.
• Can perform diagnostics on their own, but require time to pinpoint the issue
• Can function on their own, but have no management experience, or limited supervisory experience; they often tutor and instruct junior-level admins.
• Has a demeanor that is "seasoned" - they are observant and reserved, but will always be pleasant without being technical. This leads to confidence when dealing with end-users, developers, and management, and ultimately, a deep-seated sense of trust that this person will "get the job done". End-users will usually consult these folks first, but troublemakers will sometimes attempt to "game the system" and get them to do something they wouldn't (although the admin will know better and deny it). Developers will consult with this person about common issues. Management will sometimes ask for special tasks to be performed (vetted, of course, through the Guru) and they will accomplish this to their satisfaction.
• They understand the core revenue model of your business, and how this inter-relates with other positions and procedures. They can design custom solutions around this knowledge, and can find ways to decrease operational expenses. They cannot, however, create new revenue sources.

These are the admins the Guru will initially hire.

Another stereotypical impression: "This person has been around the block, and has the war wounds to prove it. If my back was against the wall, I'd put my trust in them."

A Guru:

• Has 9+ years experience.
• Can perform code-level customization of a kernel before recompile, either by reconfiguration or by writing new code
• Has seen very old kernels (2.0 or 1.3 series)
• Has experience with very difficult-to-install installations (Slackware prior to version 9, Gentoo, Linux From Scratch)
• Can do complex scripting, sometimes writing complete tools for other staffers.
• Immediately knows all potential causes of a problem and can look at each solution without additional diagnostics
• Has functioned in a supervisory or management capacity with at least one other person for at least 3 years. This means the person was hired and managed directly by them.
• Has a demeanor that borders on "happy but zen-like'. They are quiet, focused, and have an uncanny means of knowing what to say and when, while putting everyone they talk to at ease. End-users often do not notice this person because they function well at what they do, yet troublemakers are quick to fear their presence; developers will consult with this person about difficult issues; and management trusts them with staffing and employment decisions.
• They have intricate knowledge of your business process, and how your company's cash flow interacts with capital outlays, staffing, and on-going maintenance. They can find creative ways to create new revenue sources within your business model.

This is the person you want.

Another (really bad) stereotype: "Grey beard, suspenders...they must be one of those all-knowning Unix admins!"

Get a 'known' linux expert to consult on the interviews, preferably someone who has recruited linux-skilled technical staff before. Be most ruthless about filtering this person - get a charlatan in the position of signing off candidates and you will end up hiring the wrong people. Remember:

A's hire A's
B's hire C's.

You need to get an 'A' involved in the first round of hiring to get your recruitment process on track - at all costs

My advice would be to borrow a few questions from the Red Hat certification exams. These are technically Red Hat specific certs, but the knowledge applies to virtually all Linux distro's, and any competent admin should be able to answer them.

Pick a few questions from the RHCT (basic level questions), a few from the RHCE (mid level), a few from the RHCDS and RHCSS (upper level, specific knowledge), and a few from the RHCA.

You should be able to find sample questions on the internet or from training guides. You can also pull them from the pre-qualification assessment questionnaires (They can be found on the certification pages - registration might be required)

"Build" it yourself. You may start out with a junior or seasoned sysadmin. But given the right working set people start to shine.

From a commercial point of view a guru that started out as a seasoned admin in your company will nearly always be cheaper (in terms of salary), on the other hand you need to have a close look at him/her not to cause expense.

From a motivational point of view my own experience is that I was really motivated when I had my first job as a sysadmin, it started out with 1 server and I didn't even have a workstation. After about 10 months we had services running on 3 physical servers with about 20 virtualised instances (OpenVZ very lightweight) that were being used as service separation.

I wouldn't consider myself a guru (and everyone who considers him/herself a guru is to be taken with a grain of salt), but I'm pretty confident that I learned a lot more when I was on my own in that company in any given timeframe than when I am working in a team. Not because I don't like to work together but you somehow start to specialise on things because someone else is better at $topic.

Now I'm leading a team of 5 administrators (including me) and 1 developer. I consider getting a developer assigned to our team the largest success, providing the services isn't that much of an effort but having someone who enjoys developing more than administration is a major win as you can really start building larger toolchains.

So building a guru yourself may pay off. Not within a few months but in the mid to long-term. Everyone I know and consider a guru has started in harsh environments (either because working on their own, or by being assigned to tasks initially out of scope regarding their knowledge but was stilling fighting all the way thru).

A couple of quick questions to narrow down the field:

• Ask which distributions he/she has used or which are the most popular. The most popular at the moment according to DistroWatch are Ubuntu, OpenSUSE, Mint, Fedora, and Debian (which is what Ubuntu is based off of). While there are many others, the interviewee will likely cite at least one of these. Also, while its not nearly as popular (number 22 on that list), Gentoo is considered to be the one of the most "hardcore" of the distributions, but don't just rely on "he/she mentioned Gentoo so he/she must be awesome" as the only true way to know if he/she really uses it is to ask how he/she installed it or setup the kernel which is a very complicated topic. At any rate, the main point of this question is to see if he/she is familiar with several different distributions. I have found that most linux people prefer one to three, have used at least 5, and know about more.
• Ask him/her which desktop environment he/she prefers to use (or maybe to explain the difference between a few of them). The three most popular are Gnome, KDE, and XFCE. There are many others, but they are not highly used.
• Give him/her a laptop with a Linux cd (any one of the top 5 should be good), and as him/her to walk you through the installation and possibly setup. As you want someone to hire his/her own team, I also would assume that person should have great communication skills and be able to explain things to you or someone else in your company in a way that makes sense and is easyto understand. Basically check for confidence and the ability to answer questions quickly and easily.
• Also what might be good to go along with the previous point is to ask him/her to connect the newly installed laptop to a Windows shared folder and/or vice-versa. I assume this is one of the things that person will be expected to do on the job, so it is good to check that he/she knows how to do it. Again, ask how he/she is doing it to see if he/she feels confident with the system. Likely, he/she will open up the Terminal and install and use a program called Samba.
• You can also ask the person to print a document. If this printer happens to be shared on a windows computer, the challenge will be a bit harder and he/she will more likely install Samba and CUPS. Again, the only purpose is to make sure they know how to do it, are confident in their skills, and would be able to explain things easily to another member of your work.

I hope this helps a bit. While the last three are not really quick, they can be very effective. My main thought with the last ones was to casually talk to the person and get a feel for his/her confidence in linux as well as his/her communication skills. Ultimately, I do agree with ConcernedOfTunbridgeW that the best way for you may be to get a known linux expert consultant to assist with your interviews.

No matter what you do, I wish you the best of luck!