NGINX Ingress on Kubernetes doesn't use HTTPS

I am setting a Kubernetes cluster on bare metal. I used Kubeadm for the installation. To make my services accessible from outside the cluster, I installed an NGINX Ingress, using the following documentation : NGINX doc

Because I don't want to communicate with my services without TLS security, I configured the certificate thanks to cert-manager : Cert-manager doc.

$kubectl get certificate
NAME                   READY   SECRET            AGE
certificate-tls-prod   True    tls-secret-prod   3h1m
tls-secret-prod        True    tls-secret-prod   41m

The certificate generation works and are recognized by the client.

Now, I am trying to install my own Docker registry (registry:2.7.1) in the cluster. After deploying the Pod and the Service, I am trying to do a docker login from my client, but it doesn't work.

Ingress configuration :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    # nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    cert-manager.io/issuer: "letsencrypt-prod"
  namespace: default
  name: nginx-ingress
spec:
  tls:
  - hosts:
    - example.com
    secretName: tls-secret-prod
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: docker-registry
            port: 
              number: 5000

When I try to reach the service doing a simple curl :

curl -v https://example.com/

*   Trying <cluster-ip>:443...
* Connected to example.com (<cluster-ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=example.com
*  start date: Jul 13 10:03:45 2021 GMT
*  expire date: Oct 11 10:03:44 2021 GMT
*  subjectAltName: host "example.com" matched cert's "example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.77.0
> Accept: */*
> 
* Received HTTP/0.9 when not allowed
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (1) Received HTTP/0.9 when not allowed

The handshake seems correct. But, something's wrong in the end.

Log of the Docker registry pod :

http: TLS handshake error from <ingress-pod-ip>:38686: tls: first record does not look like a TLS handshake

Log of the Ingress Pod :

[error] 84#84: *40 upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: <cluster-ip>, server: example.com, request: "GET / HTTP/1.1", upstream: "http://<registry-pod-ip>:5000/", host: "example.com"

Log of Docker login from the client:

docker login https://example.com
Error response from daemon: Get "https://example.com/v2/": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

From what I understand, the problem comes from the fact that one of the component try to communicate with http instead of https. By the way, the upsteam is in http, which seems strange.
Unfortunately, I can't seem to fix this problem. I tried several annotations (which are commented) in the ingress declaration to force the use of https, but it didn't work.

I saw this issue on Github, but my apiserver doesn't have the parameter --kubelet-https=false

I would appreciate it if you had any clue that could help me.

As Thomas has mentioned in the comment:

I managed to make it works. It seems that it came from the docker registry configuration. I modified the certs and restarted the pod.

NGINX Ingress on Kubernetes doesn't use HTTPS

Related

Recent Posts