How do new package versions find their way to the LTS distribution? (Current example: OpenSSL)

I'm running several Ubuntu 18.04 machines all of which need OpenSSL.

Recently, OpenSSL reported a security problem: https://www.openssl.org/news/secadv/20200421.txt (CVE-2020-1967).

I did install OpenSSL via the Ubuntu standard packages, no custom package sources here nor did I compile it myself. So, since this is the case and since I'm using a currently maintained LTS distribution (18.04 as mentioned), I assumed that running apt update and apt upgrade would be sufficient to get the latest updates and be safe from that vulnerability. But, no. Obviously, it is more complicated.

According to https://launchpad.net/ubuntu/+source/openssl there is an OpenSSL package version for Ubuntu which reacts to the aforementioned CVE-2020-1967. However, it still contains 1.1.1f in its version name, whereas the OpenSSL version where the problem is fixed is actually 1.1.1g (according to their advice). And, more important: That package is only for Focal Fossa (20.04).

So, I thought I take the chance to learn a few things about the internals of Ubuntu and how package version make their way to my computer:

  • How come that new packages are only created for newer LTS distributions and older ones are left vulnerable?
  • How does that process work in general?
  • What is the average time between a vulnerability being publicly announced and patched packages being available via apt upgrade?

How come that new packages are only created for newer LTS distributions and older ones are left vulnerable? is simply FALSE, and let's show you how you can determine that for yourself:

First, Let's take a look at the Ubuntu CVE Tracker.

Key in your CVE number, and you will see:

  • The Ubuntu Security Team has already worked this issue.
  • Some Ubuntu releases required new packages, others did not.
  • The tracker includes the exact package version with the fix.

Second, lots of folks get confused by the Debian/Ubuntu concept of a snapshot release model. Under this model, the software in a single release (like 19.10 or 20.04) is NOT incremented by new releases...even most security releases.

  • Exceptions: There are a few exceptions, like web browsers, that always get the newest versions on older releases.

HOWEVER, "not incremented" doesn't mean "not fixed". Lots of packages get security patches after release without being upgraded to a new version. Let's say that again: Users are encouraged to update to newer packages because most users don't have the skill or patience to patch and recompile a package. Distros like Ubuntu, however, have engineers who do have that skill and patience.

  • It's risk management: Newer packages may introduce bugs and haven't been tested with the release, so most packages are patched instead.

Finally, let's build on those two elements and let's take a look at the example of OpenSSL in CVE-2020-1967:

  • The Ubuntu Security Team evaluated the vulnerability in six different releases of Ubuntu. The vulnerability was present in one of those releases.
  • The upstream release (for users) is version 1.1.1g
  • The Security Team patched the vulnerability in 20.04's package 1.1.1f. Since they added a patch instead of merging the new upstream release, they couldn't claim it was 1.1.1f anymore, nor was it 1.1.1g. It was something else, so it's called 1.1.1f-1ubuntu2. Note that the '2' in 'ubuntu2' indicates this is the third patch of that package since it came from upstream.
  • That labeling scheme for patches isn't accidental - it's carefully designed so you can see at a glance how many patches came from Debian, and how many from Ubuntu. Also, it's still alphanumerically sortable so apt will always correctly identify the highest version to install.
  • Finally, after review and testing, the patched package openssl 1.1.1f-1ubuntu2 was uploaded to 20.04 (Focal) before release. Had the patch occurred after release, it would be in the focal-security repository. This is one reason it's important to keep the -security repo in your sources, and to keep Unattended Upgrades active.