Kibana does not trust Let's Encrypt CA in Kubernetes Helm Setup

I have a k8s cluster configured with calico as networking system. I'm running slightly customized versions of helm elastic/elasticsearch and elastic/kibana with security enabled. For security, I use Let's Encrypt certificates. When starting kibana, the connection to the elasticsearch instance fails with this error

{"type":"log","@timestamp":"2021-06-01T13:09:55+00:00","tags":["debug","elasticsearch","query","data"],"pid":952,"message":"[ConnectionError]: unable to get issuer certificate"}

I can resolve this problem, by disabling certificate verification in kibana. Can anyone see, why its failing?

keystore creation

cat cert1.pem > store.pem
cat privkey1.pem >> store.pem
cat chain1.pem >> store.pem
cat fullchain1.pem >> store.pem
openssl pkcs12 -export -in store.pem -out keystore.pkcs12

values_elastic.yaml

replicas: 1
minimumMasterNodes: 1

esConfig:
   elasticsearch.yml: |
     xpack.security.transport.ssl.enabled: true
     xpack.security.transport.ssl.verification_mode: certificate
     xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
     xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
     xpack.security.http.ssl.enabled: true
     xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
     xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs-gen/keystore.pkcs12
     xpack.security.enabled: true
extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs-gen/
protocol: https
service:
  labels: {}
  labelsHeadless: {}
  type: NodePort
  nodePort: 30001
  annotations: {}
  httpPortName: http
  transportPortName: transport
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  externalTrafficPolicy: ""
clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"

values_kibana.yaml

elasticsearchHosts: "redacted its a TLD with appropriate port"

extraEnvs:
  - name: "NODE_OPTIONS"
    value: "--max-old-space-size=1800"
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: 'KIBANA_ENCRYPTION_KEY'
    valueFrom:
      secretKeyRef:
        name: kibana
        key: encryptionkey
  - name: 'LOGGING_VERBOSE'
    value: "true"


secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/kibana/config/certs-gen/


kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs-gen/privkey1.pem
      certificate: /usr/share/kibana/config/certs-gen/fullchain1.pem
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs-gen/fullchain1.pem
      verificationMode: certificate
    xpack.reporting.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    xpack.encryptedSavedObjects.encryptionKey: ${KIBANA_ENCRYPTION_KEY}

protocol: https

service:
  type: NodePort
  loadBalancerIP: ""
  port: 5601
  nodePort: 30002
  labels: {}
  annotations: {}
    # cloud.google.com/load-balancer-type: "Internal"
    # service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    # service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    # service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
    # service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true"
  loadBalancerSourceRanges: []
    # 0.0.0.0/0
  httpPortName: HTTP

kubectl get pv,pvc,nodes,pods,svc

NAME                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                 STORAGECLASS   REASON   AGE
persistentvolume/elk-data   30Gi       RWO            Retain           Bound    default/elasticsearch-master-elasticsearch-master-0                           40m

NAME                                                                STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/elasticsearch-master-elasticsearch-master-0   Bound    elk-data   30Gi       RWO                           32m

NAME                 STATUS   ROLES                  AGE   VERSION
node/kubeloadbalan   Ready    control-plane,master   28h   v1.21.1

NAME                                    READY   STATUS    RESTARTS   AGE
pod/elasticsearch-master-0              1/1     Running   0          13m
pod/kibana-kibana-7fdbd7c66d-bg5xb      0/1     Running   0          7m1s
pod/nginx-deployment-868c6bb874-tsbg4   1/1     Running   0          40m

NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
service/elasticsearch-master            NodePort    10.97.47.66     <none>        9200:30001/TCP,9300:32185/TCP   13m
service/elasticsearch-master-headless   ClusterIP   None            <none>        9200/TCP,9300/TCP               13m
service/kibana-kibana                   NodePort    10.101.78.90    <none>        5601:30002/TCP                  7m1s
service/kubernetes                      ClusterIP   10.96.0.1       <none>        443/TCP                         28h
service/nginx-service                   NodePort    10.100.231.27   <none>        80:30000/TCP                    40m

From inside the container:

kubectl exec pod/kibana-kibana-7fdbd7c66d-bg5xb -it bash

curl -k -u redacted:redacted https://redacted:30001

{
  "name" : "elasticsearch-master-0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "iXEuqB7iQ9abptIZ_Gp1yg",
  "version" : {
    "number" : "7.13.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "5ca8591c6fcdb1260ce95b08a8e023559635c6f3",
    "build_date" : "2021-05-19T22:22:26.081971330Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}


curl  -u redacted:redacted https://redacted:30001
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

That shows, that even curl cannot verify legitimacy from inside the container. That might indicate that the problem is also located on elasticsearch. The full Kibana log is a complete mess due to increased verbosity. I can post it on request.

I sank days into that. I'm burnt out and don't know where to continue.

Through discussing the possible root cause of the problem (missing CA) in the comments, I found the solution myself. The issue is caused by adding a wrong CA with

elasticsearch.ssl:
  certificateAuthorities: /usr/share/kibana/config/certs-gen/fullchain1.pem
  verificationMode: certificate

Deleting those entries resolves the problem.

Kibana does not trust Let's Encrypt CA in Kubernetes Helm Setup

Related

Recent Posts