Heartbleed: What is it and what are options to mitigate it?

This is a Canonical Question about understanding and remediating the Heartbleed security issue.

What exactly is CVE-2014-0160 AKA "Heartbleed"? What is the cause, what OSs and versions of OpenSSL are vulnerable, what are the symptoms, are there any methods to detect a successful exploit?

How can I check to see if my system is affected? How can this vulnerability be mitigated? Should I be concerned that my keys or other private data have been compromised? What other side effects should I be concerned about?

First, before freaking out, be sure that you understand whether or not this vulnerability actually applies to you. If you have a server, but have never actually had any applications using TLS, then this is not a high-priority thing for you to fix. If, on the other hand, you have ever had TLS-enabled applications, well then you're in for a treat. Read on:

What exactly is CVE-2014-0160 aka "Heartbleed"?

It's a big fricking mess, that's what it is. In short, a remotely-exploitable vulnerability was discovered in OpenSSL versions 1.0.1 through 1.0.1f through which an attacker can read certain parts of system memory. Those parts being that which hold sensitive data such as private keys, preshared keys, passwords and high valued corporate data among other things.

The bug was independently discovered by Neel Mehta of Google Security (March 21, 2014) and Finnish IT security testing firm Codenomicon (April 2, 2014).

What is the cause?

Well, errant code in OpenSSL. Here is the commit that introduced the vulnerability, and here is the commit that fixed the vulnerability. The bug showed up in December of 2011 and was patched today, April 7th, 2014.

The bug can also be seen as a symptom of a larger problem. The two related problems are (1) what process are in place to ensure errant code is not introduced to a code base, and (2) why are the protocols and extensions so complex and hard to test. Item (1) is a governance and process issue with OpenSSL and many other projects. Many developers simply resist practices such as code reviews, analysis and scanning. Item (2) is being discussed on the IETF's TLS WG. See Heartbleed / protocol complexity.

Was the errant code maliciously inserted?

I won't speculate on whether this was truly a mistake or possibly a bit of code slipped in on behalf of a bad actor. However, the person who developed the code for OpenSSL states it was inadvertent. See Man who introduced serious 'Heartbleed' security flaw denies he inserted it deliberately.

What OSs and versions of OpenSSL are vulnerable?

As mentioned above, any operating system that is using, or application that is linked against OpenSSL 1.0.1 through 1.0.1f.

What are the symptoms, are any methods to detect a successful exploit?

This is the scary part. As far as we know, there is no known way to detect whether or not this vulnerability has been exploited. It is theoretically possible that IDS signatures will be released soon that can detect this exploit, but as of this writing, those are not available.

There is evidence that Heartbleed was being actively exploited in the wild as early as November, 2013. See the EFF's Wild at Heart: Were Intelligence Agencies Using Heartbleed in November 2013? And Bloomberg reports the NSA had weaponized the exploit shortly after the vulnerability was introduced. See NSA Said to Exploit Heartbleed Bug for Intelligence for Years. However, the US Intelligence Community denies Bloomberg's claims. See IC ON THE RECORD.

How can I check to see if my system is affected?

If you are maintaining OpenSSL on your system, then you can simply issue openssl version:

$ openssl version
OpenSSL 1.0.1g 7 Apr 2014

If the distribution is maintaining OpenSSL, then you probably can't determine the version of OpenSSL due to back patching using openssl command or the package information (for example, apt-get, dpkg, yum or rpm). The back patching process used by most (all?) distributions only uses the base version number (for example, "1.0.1e"); and does not include an effective security version (for example, "1.0.1g").

There's an open question on Super User to determine the effective security version for OpenSSL and other packages when packages are backpatched. Unfortunately, there are no useful answers (other than check the distro's website). See Determine Effective Security Version when faced with Backpatching?.

As a rule of thumb: if you have ever installed one of the affected versions, and have ever run programs or services that linked against OpenSSL for TLS support, then you are vulnerable.

Where can I find a program to test for the vulnerability?

Within hours of the Heartbleed announcement, several people on the internet had publicized publicly-accessible web applications that supposedly could be used to check a server for the presence of this vulnerability. As of this writing, I have not reviewed any, so I won't further publicize their applications. They can be found relatively easily with the help of your preferred search engine.

How is this vulnerability mitigated?

Upgrade to a non-vulnerable version and reset or re-secure vulnerable data. As noted on the Heartbleed site, appropriate response steps are broadly:

Patch vulnerable systems.
Regenerate new private keys.
Submit new CSR to your CA.
Obtain and install new signed certificate.
Invalidate session keys and cookies
Reset passwords and shared secrets
Revoke old certificates.

For a more detailed analysis and answer, see What should a website operator do about the Heartbleed OpenSSL exploit? on the Security Stack Exchange.

Should I be concerned that my keys or other private data have been compromised? What other side effects should I be concerned about?

Absolutely. Systems Administrators need to assume that their servers which used vulnerable OpenSSL versions are indeed compromised and respond accordingly.

Shortly after the vulnerability was disclosed, Cloudfare offered a challenge to see if a server's private key could be recovered in practice. The challenge was independently won by Fedor Indutny and Ilkka Mattila. See The Heartbleed Challenge.

Where can I find more information?

Link dump, for those looking for more details:

OpenSSL SECADV 2014047
CVE-2014-0160
Heartbleed
Ubuntu announcement
RHEL announcement
Official OpenSSL announcement

A rather detailed timeline of the disclosure events can be found at Heartbleed disclosure timeline: who knew what and when.

If you are a programmer and are interested in various programming tricks like detecting a Heartbleed attack through OpenSSL's msg_cb callback, then see OpenSSL's Security Advisory 2014047.

A simple explanation of the bug, by XKCD:

XKCD 1354

Ubuntu 12.04, 12.10, and 13.10

Ubuntu has issued USN-2165-1, which states that updated packages are now available in the archives. Run the following two commands to grab the fix.

sudo apt-get update
sudo apt-get upgrade

Ubuntu 14.04

I have uploaded a Debian package containing the new release (1.0.1g) to a PPA I have set up for this purpose. These three commands will add my PPA to your system, update the list of available packages, and upgrade everything:

sudo add-apt-repository ppa:george-edison55/openssl-heartbleed-fix
sudo apt-get update
sudo apt-get upgrade

_{Note: the PPA also provides packages for Ubuntu 12.04 and 13.10, just in case you prefer to actually run the new version (1.0.1g) instead of just using the patched versions in the archives.}

Ubuntu 10.04

This is an LTS Version, the server version is still supported and receives security updates. But the heartbleed vulnerability did not affect the openssl package of a standard installation of ubuntu 10.04, because the version is below 1.0.1.

The desktop version has reached end of life and needs to be upgraded / reinstalled.

Ubuntu 13.04 and other outdated versions

Ubuntu 13.04 had a very short support cycle which you might not expect. It has reached end of life already and does not receive security updates any more. It should long have been upgraded. If still someone is using it, please upgrade now, either from scratch or it can be upgraded non-destructive to 13.10 following this easy procedure: http://www.tecmint.com/upgrade-ubuntu-13-04-raring-ringtail-to-ubuntu-13-10-saucy-salamander/ After the upgrade the system receives the heartbleed patch from 13.10.

For all other outdated ubuntu versions it means basically a fresh install is necessary.