How To Deal With Frequent And Constant PHP/MySQL Version Updates
The Problem:
I know that many people are not so happy about frequent version changes. Especially when latest version of PHP or MySQL that you are stick to is no longer available in your repository (in my situation it is REMI) but you have to install a new server with some PHP/MySQL packages of a version a little bit lower (older/earlier) from thus that available in repository.
Most of the time we are using the latest version of the typical Apache, PHP, MySQL on top of the Centos 5.8 GNU/Linux x86_64 operating system.
But now it takes a lot of time for our QA team to test all of our projects for compatibility with the newer versions that comes so fast that at the time we have green light from QA team to update PHP and/or MySQL and/or install a new server with that particular versions, we discover that it already become obsolete and now it is superseded with more fresh version.
This is hearts especially when McAfee PCI Compliance says that our sites using potentially dangerous version of PHP and force us to upgrade all the server with the newer version of PHP.
At the current moment our default working environment consist of thus guys:
OS:
CentOS release 5.8 (Final) Linux censored.example.com 2.6.18-308.4.1.el5 #1 SMP Tue Apr 17 17:08:00 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
Apache:
Server version: Apache/2.2.3 Server built: Feb 23 2012 21:16:56
PHP:
PHP 5.3.13 (cli) (built: May 9 2012 16:20:57) Copyright (c) 1997-2012 The PHP Group Zend Engine v2.3.0, Copyright (c) 1998-2012 Zend Technologies with eAccelerator v0.9.6.1, Copyright (c) 2004-2010 eAccelerator, by eAccelerator
MySQL:
mysql Ver 14.14 Distrib 5.5.23, for Linux (x86_64) using readline 5.1
Possible Solutions:
Local Repository/Old Versions Mirror Just make custom repository snapshot of latest installed packages so we could setup/install new servers with same packages even if they are no longer supported before we will be ready to move on. Pros & Cons: Easy installation, using same YUM commands but won't it break dependencies with other system packages that may be updated are there hidden water stones in that way?
Compile From Source Forget about YUM and RPM and go back to good old days with GCC compiler and dependency hells by using statically compiled versions. Pros & Cons: No more dependent on repository versions changes (or still do I in some indirect way?) but will have to obtain all the source code manually and have some sort of management of that stuff.
Build And Use Custom Image Just build and configure once an suitable server, take its snapshot (either as virtual machine or either its physical HD iso image) and than just expand its image on the new physical server or import virtual machine into visualization host. Pros & Cons: Surely no more headaches with version changes and dependencies but how it is reliable? Yes it is a peace of cake if we are talking about virtual machines but some time we have to deploy projects on a physical machines we have nothing but SSH access.
Accept Updates If It's Only Minor Version Change Do not worry to much about bug fixes and version changes due to critical security issues fixed that rises minor number. Just update to them ASAP. Pros & Cons: Surely McAfee PCI Compliance will be happy and so will do our clients but doesn't it is dangerous? What if it was a minor change in default PHP configuration values that we didn't overridden? Won't it be a possible total disaster?
Any suggestion dear experts?
Solution 1:
Ok - so let's boil this down:
The primary issue you're facing is that you need to run a newer version of PHP/MySQL, but by the time your QA team has tested the software on the new system, that version is out of date.
That's not a failing of your team; that's a failing of the QA team.
Your QA team should have automated test cases. Your developers should be providing test modules as they go along. E.g.:
- Developer builds pro-rated invoicing system
- Developer provides a test interface for pro-rating (i.e. accepts all the details needed to test the system, but does not actually generate an invoice. Or maybe it does, but in a staging system).
- QA team devise a series of tests to run against the routine. What is the input, and what is the correct outcome? E.g. $50/month pro-rated for half a month == $25.
This last step should be fully automated. It should take a matter of minutes or hours (depending on the complexity of the software or how many tests there are) to determine any errors. E.G.
- PHP changes its floating point maths from normal rounding to cieling (always round up)
- Test case of pro-rating at $5/month for half a month
- Correct answer: $2.50, but test case returns $3. Automatic fail.
Now, I know that none of this really answers your question, so if your QA team can't be fixed, then what are our other options:
There's no one correct answer to this. Personally, what we do with our software is your last option - we accept all updated and service packs to our runtime environments, but we do not change versions without a thorough and complete QA analysis. Although, the software we use in our runtimes (which is MSSQL and another back-end, not PHP) only release new versions every 18 months to 2 years, with updates and service packs inbetween, so there's plenty of time between major rollouts.
After QA tests on our staging systems, we deploy the update to our dogfood systems first. If we don't find any major issues, we roll the runtime updates out to our hosted systems. If we still don't have any major issues, we release notices to our self-hosted clients that they can install the updates if they wish.
Personally I cringe every time I get a pre-packaged virtual machine. Even if it's totally up to date (they never are, as you've pointed out) it's something that I then need to work on integrating into our network. And sometimes they ship with horribly out-dated operating systems or runtimes.
Solution 2:
You have stated your question as follows:
Periodically, critical security flaws are found in components of your server stack, including PHP and MySQL.
When such flaws are discovered, the developers of PHP and MySQL release new versions. In the interest of security, old/insecure versions may no longer be available in the repositories that you use.
Your web application must be PCI compliant. When you use old, insecure versions of PHP and MySQL, your web application fails PCI security scans that must pass in order to achieve compliance.
You are never able to use the current version with all security fixes, because your QA team takes "a lot of time" (apparently months) to give the green light to use each security release of PHP and MySQL. In fact, they wait so long to approve security updates that you are never fully patched.
You then proceed to explore various possibilities for making it easier to install old versions of PHP and MySQL with known vulnerabilities. However, items #1-3 above are not negotiable.
Item #4, on the other hand, is negotiable. Ergo, your company's QA team is the problem. Security updates incrementing the version number by 0.0.1 (e.g. from PHP v5.4.2 to v5.4.3) are very unlikely to break your application.
You should install the new version in your test environment, run your automated tests, do minimal manual testing, and then roll it. As with any change, you need to have a back-out plan in place in case of unexpected problems, but... come on, you're talking about patches for known exploits! The security risk that you take by not patching (which, by the way, includes guaranteed failure of your PCI compliance scans) far exceeds any risk you take by patching.
Feature upgrades (e.g. from PHP v5.3.x to v5.4.x) may require more thorough testing, and for good reason: for example, 5.4.0 removed various legacy features and some old applications might need to be updated accordingly. Fortunately, you actually do get months to test these releases. This is exactly why security updates are provided concurrently for both the current version and the previous version.
It sounds an awful lot like your QA team is confusing minor security updates with actual feature upgrades.
Executive Summary:
- Security updates that do not add/remove features must be applied immediately with minimal testing. It is unacceptable for your company's processes to subject minor security updates to bureaucratic delays, unless actual problems are found in the initial testing.
- When it comes to feature releases that have a much higher probability of breaking legacy applications, your QA team can continue to take its sweet time.