Best Practices for updating a previously unmaintained server RHEL5.7

A new RedHat EL5.6 server has been recently put under my care. It is immediately obvious that, for the previous 12 months, little to no attention has been given to any sort of package updates.

Typically I am of the mindset of if it isn't broke - don't fix it. However, after registering the server with RHN, and also using the yum-security plugin to check security updates, there are just over 1100 "security" updates available.

Has anybody had a similar situation? I am reluctant to just update everything, as I like to know what is being updated and whether or not it has the potential to impact anything running on the box (this is a production server). However, it also looks like keeping in line with this practice would require me to go through 1100 package errata line by line. Is there a more efficient solution?

Solution 1:

Generally speaking speaking security updates are considered to be somewhat safe, particularly for a distribution with goals like RedHat. Their core focus is creating an operating environment that is consistent. As such the maintainers tend to pick versions of packages and stick with them for the long haul. To see what I mean look at the versions of such packages like kernel, python, perl, and httpd. What they also do is backport security patches from the upstream developers. So if a security vulnerability is found for all versions of Apache httpd 2.2.x then the Apache foundation may release version 2.2.40 with the fix, but RedHat will roll the patch locally and release httpd-2.2.3-80 with the fix.

Also keep in mind that you're currently talking about a RHEL5.7 system, the current release is 5.9. Some software vendors will only support certain subreleases. I've recently come across one piece of software, for example, that the vendor says only works on 5.4. That doesn't mean it won't run on 5.9, but it may mean that they won't provide any support it if doesn't work.

There are also concerns with doing mass updates of a system that hasn't been patched in such a long time. The biggest one that I've come across is actually more of a configuration management problem that can just be exacerbated by big updates. Sometimes a config file is changed but the administrator never restarts the service. This means that the config on disk has never been tested, and the running config may no longer exist. So if the service gets restarted, which will happen once you apply the kernel updates, it may not actually restart. Or it may act different once it restarts.

My advice, would be to do the updates, but be smart about it.

• Plan it out during a maintenance window. If nothing else the server will require restarting, there have been a number of kernel updates and you will have to reboot to apply them.
• Make sure to take a full backup before doing anything. This could be snapshotting, if this is a VM, triggering a full backup on whatever your tool is, tarring up / (to another system), taking a dd image of the drives, whatever. Just so long as it's something you can restore from.
• Plan out how you apply the updates. You don't want to just throw a yum update -y at it and walk away. For all of the good things that yum does do it does not order when it applies updates according to the dependencies. This has caused problems in the past. I always run yum clean all && yum update -y yum && yum update -y glibc && yum update. That tends to take care of most of the potential ordering issues.

This may also be a great time to replatform. We've had RHEL6 for quite a while now. Depending on what this server does, it may make sense to just let this one run as is while you bring up a new instance in parallel. Then once it's installed you can copy all the data over, test the services, and perform the cut over. This will also give you the chance to know, from the ground up, that the system is standardized, clean, well documented, and all that jazz.

No matter what you do, I feel it's pretty important that you get yourself up to a current system. You just need to make sure to do it in a way that lets you trust your work and the finished product.

Solution 2:

In my experience RHEL does not break backwards compatibility in same-release updates.

that, however will not extend to anything that has been installed externally to rpm.

You could use rpm -qf to find the files you suspect are compiled external, if it returns "not owned by any package" then you might have issues in your upgrade.

I'd take an image of the server and do the upgrade, I'm a little more devil-may-care than most however.