How can a Linux Administrator improve their shell scripting and automation skills?

In my organization, I work with a group of NOC staff, budding junior engineers and a handful of senior engineers; all with a focus on Linux. One interesting step in the way the company grows talent is that there's a path from the NOC to the senior engineering ranks. Viewing the talent pool as a relative newcomer, I see that there's a split in the skill sets that tends to grow over time...

• There are engineers who know one or several particular technologies well and are constantly immersed... e.g. MySQL, firewalls, SAN storage, load balancers...
• There are others who are generalists and can navigate multiple technologies.
• All learn enough Linux (commands, processes) to do what they need and use on a daily basis.

A differentiating factor between some of the staff is how well they embrace scripting, automation and configuration management methodologies. For instance, we have two engineers who do the bulk of Amazon AWS CloudFormation work, and another who handles most of the Puppet infrastructure. Perhaps a quarter of the engineers are adept at BASH shell scripting.

Looking at this in the context of the incredibly high demand for DevOps skills in the job market, I'm curious how other organizations foster the development of these skills and grow their internal talent. Scripting doesn't seem like a particularly-teachable concept.

• How does a sysadmin improve their shell scripting?
• Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?
• Are we simply to assume that some people will be left behind as these technologies evolve? Is that okay?

I have the benefit of understanding the size and complexity of your environment. Seeing as you work for a cloud/hosting provider, it's safe to assume that you have a large number of small-medium sized environments (10-100 servers). There are certainly daily tasks that are done by the jr. engineers and NOC staff that are repetitive (creating user accounts, configuring backup agents, etc). Similarly, there are probably some manual things that are done by the sr. engineers like installing ESXi on new hardware or configuring things like MPIO or installing VMware modules for specific sets of hardware. All of these things can and should be automated.

If your staff is capable of carrying out the bulk of their workload without automating, then you're overstaffed in my opinion. Any IT staff that can work a full day that consists of mostly manual processes has no motivation to automate. Why learn a new skill that isn't viewed as necessary and might even be scary? After all, necessity is the mother if innovation.

So, at some point in your organization, you will grow to a size where you'll flounder and fall apart, or you'll start automating almost everything and excel. Certainly, the senior engineers should be leading the charge here, and maybe even working with the junior engineers and NOC staff to automate some of their workload. This gives the jr. engineers the opportunity to have the framework of many scripts to work with, which they can tweak for each tenant and new hardware revision as necessary. This removes the daunting thought of "Oh my god, where do I even start?" from the equation and gives them a jumpstart to solving a real problem. Which brings me to my final point. Books and examples are well and good, but there's nothing that can replace the feeling of accomplishment of solving an actual problem that they face. Give them a goal, like all new servers for tenant x should have certain ESXi modules installed, and then work with them to accomplish it. Then adapt the script to work in a multitenant environment.

How does a sysadmin improve their shell scripting?

By needing to, as described above.

Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?

Sure, there are plenty of organizations that cannot or will not shift to the DevOps methodology. They're seeming to be more and more boring options, but they're options nonetheless.

Are we simply to assume that some people will be left behind as these technologies evolve?

As with any new technology - yes.

tl;dr You'll never have anyone really invest in learning it until they see the value in it. If they can accomplish their daily tasks manually, then you're overstaffed and there's no incentive.

• How does a sysadmin improve their shell scripting?

Practice, mixed with drive. It sounds trite, but you have to want to get better, in addition to practice. If you don't truly enjoy scripting, you can be forced to do it for years when you have to and never really get good at it. If you don't want to get better, you could sit next to the best scripter in the world every day at work and not pick up on a fraction of the skill that you could have.

I know those people that, despite working in IT, stubbornly refuse to learn any sort of scripting. There will soon be no place for those people in this industry. They're part of a dying generation.

(I'm not talking about old people, I mean that figuratively. :P)

• Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?

Nope. Every thing that they do can be and eventually will be automated away.

I would contend that perhaps we never should have called them 'engineers' anyway. It's bad enough that the IT industry appropriated the word 'engineer' for ourselves, which in my opinion is kind of insulting to the actual engineers that spent years in higher education programs and getting legal certifications so that they could design bridges, skyscrapers, hadron colliders, etc... those are the real engineers.

But there is a similarity... If you want to call yourself an 'engineer' in the IT industry, then that at least means you create things. You are inventive and you connect the dots in new ways that no one ever thought of before. You build things that no one else knew how valuable it would be until you made it.

If you don't code or script, then there is no way you do much with computers besides just maintain them, and maybe install a software package or two. Maybe throw a new hard drive into the ol' MSA. And in that case, I'd call you an administrator, sure, but not necessarily an engineer. And I'd say much of your job is in jeopardy of being automated away.

• Are we simply to assume that some people will be left behind as these technologies evolve?

The market will adapt. It may be that some people will not be making 6-figure salaries when they don't actually deserve them, which happens quite a bit in this industry.

I find that creativity, and not just coding/scripting skill, is a key factor. It's that creativity that you need to say to yourself, "Oh, hey, I could automate this!" and then the skill only comes into play after that. If you find yourself scripting something only after your boss tells you to, then you might not have that drive or that creativity I was talking about... and those are two qualities that are very hard, maybe impossible, to teach.

How does a sysadmin improve their shell scripting?

How does one get better at anything? Read books, attend classes, and then apply the principles learned. (Or a combination of the methods.) This is over simplified intentionally since there isn't anything special about learning scripting over learning how to cook or how to repair a car.

Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?

This is hard to answer within the scope of this site (where there is a requirement for clear/defined answers to questions asked.) We can predict that it will, but there are problems with the DevOps model. I feel that it's very hard for one person to be extremely proficient in both disciplines. The cost savings of a 2-for-1 employee is very attractive to businesses right now, but it's hard to say if this trend is here to stay. It certainly is for the short term.

Are we simply to assume that some people will be left behind as these technologies evolve?

At the current rate of how things are going, yes. Most of you are likely observing it in your own workplaces. You should definitely be keeping up with job listings and know what the market is currently demanding. (There's a lot of job listings for Hadoop in your area? Learn Hadoop.) If you don't keep up with the market, you are risking being left behind.

One generally doesn't send junior engineers into a complex production environment that is mission critical. You have senior engineers for that. Junior ranks should be allowed to work in dev / test sandboxes.

If you need an engineer for Technology X and want to fill the role internally, find someone willing to learn it, find structured training and combine the two.

Figure out what skills you need in a department. Find someone willing to learn them. Teach / Hand out money for training.