Best practices for keeping EC2 Ubuntu machines updated

Following the recent heartbleed vulnerability of OpenSSL, I would like to keep my EC2 machines updated on a regular basis. The naive approach would be setting an hourly cron job for security updates (sudo apt-get update && sudo unattended-upgrade).

Are there any risks of doing that? Is there a recommended update mechanism for EC2 machines?

Solution 1:

The unattended-upgrades package is the standard way to automatically apply important bug fixes and security patches in Ubuntu.

I recommend installing this on every Ubuntu system:

sudo apt-get update &&
sudo apt-get install unattended-upgrades

You don't need to create your own cron job. The package installs one for you.

You can edit the default configuration if you'd like to alter its behavior: https://help.ubuntu.com/lts/serverguide/automatic-updates.html

Solution 2:

We have been using unattended-upgrades from 2015 to 2020 with no issues. We have a small setup (on DigitalOcean) with:

• nginx
• mysql-server
• php5-fpm php5-curl php5-mysql

Based on good past performance, doing updates this way feels safer than not doing it. But that's not necessarily a guarantee for the future!

This might not be such a good idea for apache, based on the reports of other users, and my past experience of apache updates. [See above, and here]

With unattended-upgrades, manual intervention will still be required when the release approaches EOL.

(Aside from the question: In my experience with TWiki, WordPress and Jenkins, keeping these apps up-to-date is actually a greater concern than the OS itself, although of course we should really do both. For peace-of-mind, you could sandbox Internet-facing apps as non-root processes running inside a Docker container.)

But since you asked about best practice, the primary approach recommended in AWS documentation is:

• Create and start new instances to replace your current online instances. Then delete the current instances.

  The new instances will have the latest set of security patches installed during setup.

(February 2020)

This can be done as part of a blue-green deployment strategy. The advantage here is that you can run tests against your new server before switching traffic over. If your tests are thorough, then in theory your updates can be fully-automated, verified before going live, and with no downtime.

Other advantages:

• The tests can give you advanced warning if human attention is required (as opposed to unattended-upgrades, when the warnings only come from your users once the problem is live!)

• If your system does get compromised, or you decide to switch providers, this approach should make it easy to roll out a fresh deployment. Your deployment strategy is scripted, rather than ancient memory.

But of course there is more setup required for this approach than simply installing unattended-upgrades, and more complexity, so there is still room for error.

AWS does also mention performing the "Update Dependencies stack command", which seems to be their official way of doing something similar to unattended-upgrades. It seems that can be triggered from their Instances interface, but I'm not sure if it can be automated.