How do you apply development practices like version control, testing and continuous integration/deployment to system administration?
Solution 1:
Short answer is "OS deployment management", "Configuration Management" and "Software Packaging". Long answer follows.
I'd like to add to Daniel Pittman's reply with a breakdown of what forms a "system" in systems administration.
A system or an environment is going to comprise of:
- Servers
- Operating System
- Configuration
- Vendor packages; and
- Local packages
Enveloping these will be processes such as:
- OS deployment or imaging
- Configuration management
- Software package management
- Auditing/logging
- Monitoring
- Backups
And you'd like to have these combine together to assist you with achieving non-functional goals such as:
- Repeatability
- Maintainability
- Measurability
- Performance
- Trackability
- Testability
- Changeability
This is a quick brain dump. I am sure more could be added to all lists.
Your question touches on a number of these without using the specific words. For example, you want to be able to deploy easily and revert back i.e. want maintainability; you want to do it in a test environment and test till it passes i.e. repeatability, testability and measurability; you're thinking of putting vm images in version control because you'd like repeatability of OS and configuration deployments.
There are plenty of tools to assist you with this, some of them mentioned by Daniel. Some others are:
- Kickstarts (RedHat based), Preseed (Debian based), WDS (MS Windows) for deploying known OS environments
- Spacewalk/Satellite (RedHat based), Group Policies (MS Windows) for configuration and package management
- YUM and APT packaging systems for generating, deploying, upgrading and removing packages (sets of binaries, data and configuration comprising a piece of software)
- Nagios, OpenNMS and SCOM for monitoring
- Amanda, Bacula and Windows Backup Server for backups
- Munin, PCP and Hyperic for performance monitoring
- CVS, SVN, GIT or Bazaar for Version Control
- Hudson and Jenkins for build management
- Selenium and Robot for testing
- Bugzilla, Request Tracker and Jira for recording, communication and tracking
Again, this isn't a comprehensive list, but something that I keep in my head to guide me and hopefully it'll help you out too.
Solution 2:
Disclaimer: I am one of the developers of Puppet.
The obvious way is to just apply the concepts: define a cycle of development/test/production, and push changes through them. Use version control to track systems.
In short order, starting down that path leads to discovering that you really want tools that automate those things - essentially, that you want to automate systems administration, so that you don't use those techniques on machines, you use them on the system that manages the machines.
Tools like Chef, Puppet, Salt, and CFEngine are all popular tools to address that second need. They work in the general direction of turning systems administration into a central solution that you can version control and test.
The DevOps movement is another source of good information on how to do this. While the precept is a better cooperation between Developers and Operations staff, it also tends down this same direction.
Solution 3:
In a Windows world, these issues around application lifecycle management are being addressed with System Center 2012.
In System Center Virtual Machine Manager (SCVMM), services are defined using 'Service Templates' (e.g. a classic three-tier service) and execution environments are defined as 'Clouds' (e.g. development, staging, production). Service Templates can be versioned and rolled out (in an automated fashion) to different Clouds. Under the covers, SCVMM does the work to provision, deploy and configure the virtualised hardware (VMs etc) and software (OS, app components etc).
System Center Service Manager is the piece that ties this together from a process point-of-view. For example, the issue management and change control.