improve our deployment strategy

My advice for doing releases is to have Feature Releases and Maintenance Releases. Feature Releases would be the releases that get new features. These get added to your subversion trunk. When you think these are feature complete, you branch these into a release branch. Once your QA process is happy with this release, you tag the release and deploy the code to your servers.

Now, when you get a bug report, you commit this fix to the branch and port it to the trunk. When you're happy with the number of bugs fixed, you can tag and deploy a Maintenance Release.

It's important that you have a branch of your live code base (or have the ability to create one by knowing the live revision) that is separate from your development branch, so that you have the ability to deploy fixes to your live code without having to deploy new features or untested code.

I would recommend using your distribution's native packaging system for deploying new code. If you have a package that contains all your code base, you know all your code has been deployed in a sort of atomic operation, you can see what version is installed at a glance, can verify your code base using your packages checksumming. Rolling back is just a case of installing the previously installed version of the package.

The only roadblock I can see to you implementing this is that you appear to have multiple copies of the code base for different customers running on a single server. I would attempt to arrange your code so that all customers run off the same files and don't use copies. I don't know how easy that would be for you, but reducing the number of copies you have to deal with will massively reduce your headache.

I'm assuming that as you mentioned LAMP, you're using PHP or another scripting language, which doesn't require a compilation process. This means you're probably missing out on a wonderful process called Continuous Integration. What this basically means is that your code is continuously being tested to make sure it's still in a releasable state. Every time someone checks in new code, a process takes it and runs the build and testing process. With a compiled language you'd usually use this to make sure the code still compiled. With every language you should take the opportunity to run unit tests (your code is in testable units isn't it?) and integration tests. For websites, a good tool to test integration tests is Selenium. In our Java builds, we also measure code coverage and code metrics to see how we progress over time. The best CI server we've found for Java is Hudson, but something like buildbot might work better for other languages. You can build packages using your CI server.

We started using Puppet (flagship product of Reductive Labs). It is a Ruby-based framework for automating sys-admin jobs. I was at puppetcamp a couple of weeks ago and here are the video links:

Luke Kanies Presenting - Puppet Intro

Also, if you'd like to see all the presentations made at puppetcamp in san francisco, this is the link:

Presentations made on how others used Puppet

Enjoy.