Failed tls handshake. Does not contain any IP SANs

I'm trying to set up logstash forwarder, but I have issues with making a proper secure channel. Trying to configure this with two ubuntu (server 14.04) machines running in virtualbox. They are 100% clean (not touched hosts file or installed any other packages other than the required java, ngix, elastisearch, etc, for logstash)

I do not believe this is a logstash issue, but improper handling of certificates or something not set correct either on the logstash ubuntu or forwarder machine.

I generated the keys:

sudo openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

My input conf on logstash server:

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

Keys were copied to forwarder host, which has the following config.

{
  "network": {
    "servers": [ "192.168.2.107:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
    "ssl key": "/etc/pki/tls/certs/logstash-forwarder.key"
  },
  "files": [
    {
      "paths": [
        "/var/log/syslog",
        "/var/log/auth.log"
       ],
      "fields": { "type": "syslog" }
    }
   ]
}

With logstash server running, I 'sudo service logstash-forwarder start' on the forwarder machine, giving me the following repeated error:

Jul  9 05:06:21 ubuntu logstash-forwarder[1374]: 2014/07/09 05:06:21.589762 Connecting to [192.168.2.107]:5000 (192.168.2.107)
Jul  9 05:06:21 ubuntu logstash-forwarder[1374]: 2014/07/09 05:06:21.595105 Failed to tls handshake with 192.168.2.107 x509: cannot validate certificate for 192.168.2.107 because it doesn't contain any IP SANs
Jul  9 05:06:22 ubuntu logstash-forwarder[1374]: 2014/07/09 05:06:22.595971 Connecting to [192.168.2.107]:5000 (192.168.2.107)
Jul  9 05:06:22 ubuntu logstash-forwarder[1374]: 2014/07/09 05:06:22.602024 Failed to tls handshake with 192.168.2.107 x509: cannot validate certificate for 192.168.2.107 because it doesn't contain any IP SANs

As I mentioned earlier, I do not believe this is a logstash issue, but certificate/machine config issue. Problem is, I can't seem to solve it. Hopefully some clever minds here can help me out?

Thanks

... Failed to tls handshake with 192.168.2.107 x509: cannot validate certificate for 192.168.2.107 because it doesn't contain any IP SANs

SSL needs identification of the peer, otherwise your connection might be against a man-in-the-middle which decrypts + sniffs/modifies the data and then forwards them encrypted again to the real target. Identification is done with x509 certificates which need to be validated against a trusted CA and which need to identify the target you want to connect to.

Usually the target is given as a hostname and this is checked against the subject and subject alternative names of the certificate. In this case your target is a IP. The validate the certifcate successfully the IP must be given n the certificate inside the subject alternative names section, but not as an DNS entry (e.g. hostname) but instead as IP.

So what you need to is:

1. Edit your /etc/ssl/openssl.cnf on the logstash host - add subjectAltName = IP:192.168.2.107 in [v3_ca] section.

2. Recreate the certificate

3. Copy the cert and key to both hosts

PS Consider adding -days 365 or more to the certificate creation commandline as the default certificate validity is just 30 days and you probably do not want to recreate it every month..

There is a script for creating proper certs for lumberjack that was mentioned on a logstash github ticket: SSL handshake fails because IP SANs are missing

Download the file:

curl -O https://raw.githubusercontent.com/driskell/log-courier/1.x/src/lc-tlscert/lc-tlscert.go

...build it:

go build lc-tlscert.go

..and run:

./lc-tlscert 
Specify the Common Name for the certificate. The common name
can be anything, but is usually set to the server's primary
DNS name. Even if you plan to connect via IP address you
should specify the DNS name here.

Common name: you_domain_or_whatever

The next step is to add any additional DNS names and IP
addresses that clients may use to connect to the server. If
you plan to connect to the server via IP address and not DNS
then you must specify those IP addresses here.
When you are finished, just press enter.

DNS or IP address 1: 172.17.42.1 (th ip address to trust)
DNS or IP address 2: 

How long should the certificate be valid for? A year (365
days) is usual but requires the certificate to be regenerated
within a year or the certificate will cease working.

Number of days: 3650
Common name: what_ever
DNS SANs:
    None
IP SANs:
    172.17.42.1

The certificate can now be generated
Press any key to begin generating the self-signed certificate.

Successfully generated certificate
    Certificate: selfsigned.crt
    Private Key: selfsigned.key

Copy and paste the following into your Log Courier
configuration, adjusting paths as necessary:
    "transport": "tls",
    "ssl ca":    "path/to/selfsigned.crt",

Copy and paste the following into your LogStash configuration, 
adjusting paths as necessary:
    ssl_certificate => "path/to/selfsigned.crt",
    ssl_key         => "path/to/selfsigned.key",

I had a real issue with this. I'm not using logstash, I was simply trying to get IP SANs to work with docker tls. I would create the certificate as described in the docker article on https (https://docs.docker.com/articles/https/), then when I would connect from a docker client:

docker --tlsverify  -H tcp://127.0.0.1:2376 version

I would get this error :

...
FATA[0000] An error occurred trying to connect: Get https://127.0.0.1:2376/v1.16/version: \
x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs 

which was driving me crazy. I admit, I stumble around in all things openssl, so, everybody might already know what I discovered. The subjectAltName example here (and every where else) shows updating the openssl.cnf file. I couldn't get that to work. I did a locate on the openssl.cnf, copy it to a local directory, then made the changes to it. When I examined the cert it did not contain the extension:

openssl x509 -noout -text -in server-cert.pem

The command being used to create that cert is here (from the docker article):

openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem \
    -CAcreateserial -out server-cert.pem

You can't add a -config openssl.cnf line to this command, it is not valid. Nor can you copy the openssl.cnf file to the current directory, modify it, and hope to get it to work that way. A few lines later I noticed that the 'client' cert uses an -extfile extfile.cnf. So, I tried this:

echo subjectAltName = IP:127.0.0.1 > extfile.cnf
openssl x509 -req -days 365 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
   -out server-cert.pem -extfile extfile.cnf

and that fixed it. So, for whatever reason my version of openssl wasn't allowing me to modify the openssl.cnf file, but, I could specify the subjectAltName like this. Works great!

You can specify any number of IP addresses, like IP:127.0.0.1,IP:127.0.1.1 (non localhost as well).

As of OpenSSL 1.1.1, providing subjectAltName directly on command line becomes much easier, with the introduction of the -addext flag to openssl req

Example:

export HOST="my.host"
export IP="127.0.0.1"
openssl req -newkey rsa:4096 -nodes -keyout ${HOST}.key -x509 -days 365 -out ${HOST}.crt -addext 'subjectAltName = IP:${IP}' -subj '/C=US/ST=CA/L=SanFrancisco/O=MyCompany/OU=RND/CN=${HOST}/'

Inspired by link