Ingress Nginx SSL 503 Error

Service.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    io.kompose.service: grafana
  name: grafana
spec:
  ports:
    - name: "3000"
      port: 3000
      targetPort: 3000
  selector:
    io.kompose.service: grafana
status:
  loadBalancer: {}

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.22.0 (HEAD)
  creationTimestamp: null
  labels:
    io.kompose.service: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      io.kompose.service: grafana
  strategy: {}
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert
        kompose.version: 1.22.0 (HEAD)
      creationTimestamp: null
      labels:
        io.kompose.service: grafana
    spec:
      containers:
        - env:
            - name: GF_SERVER_DOMAIN
              #              value: "testing.esamir.com"
              value: "direct.esamir.com"
            - name: GF_SERVER_ROOT_URL
              value: "%(protocol)s://%(domain)s:%(http_port)s/"
            - name: GF_SERVER_SERVE_FROM_SUB_PATH
              value: "true"
            - name: GF_DATABASE_URL
              valueFrom:
                configMapKeyRef:
                  name: grafanaconfig
                  key: url

          image: grafana/grafana:7.4.0-ubuntu
          name: grafana
          ports:
            - containerPort: 3000
          resources: {}
      restartPolicy: Always
status: {}

ingress-nginx config:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: hello-kubernetes-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: testing1.com
      http:
        paths:
          - backend:
              serviceName: hello-kubernetes-second
              servicePort: 80
    - host: testing2.com
      http:
        paths:
          - backend:
              serviceName: dashboard
              servicePort: 3000

if I connect to http or https on testing1.com which is running a simply hello-world kubernetes app. Everything works flawlessly.

If I connect to testin2.com on https or http, I get the 503 error. "503 Service Temporarily Unavailable"

Looking at the logs, I found this error: "2021-02-09 15:34:25.309 PSTError obtaining Endpoints for Service "test/dashboard": no object matching key "test/dashboard" in local store"

For reference all of my manifests are deployed under the test namespace in K8.

Most of the references I found online talk about ensuring the service has a valid endpoint.

kubectl get endpoints --namespace test

NAME                                       ENDPOINTS                                      AGE
grafana                                    10.68.5.23:3000                                6h4m
hello-kubernetes-second                    10.68.3.9:8080,10.68.5.5:8080,10.68.6.6:8080   6h22m
nginx-ingress-nginx-controller             10.68.5.8:443,10.68.5.8:80                     6h20m
nginx-ingress-nginx-controller-admission   10.68.5.8:8443                                 6h20m

kubectl describe svc grafana -n test

Namespace:         test
Labels:            io.kompose.service=grafana
Annotations:       <none>
Selector:          io.kompose.service=grafana
Type:              ClusterIP
IP:                10.71.248.111
Port:              3000  3000/TCP
TargetPort:        3000/TCP
Endpoints:         10.68.5.23:3000
Session Affinity:  None
Events:            <none>

As far as I can tell, my services do have a valid endpoint. What is it that I'm missing?

Solution 1:

As it was mention in comment section, Root cause of the issue was Ingress misconfiguration.

One of the Ingress paths was pointing to non existing service - dashboard. Once it was changed to grafana, ingress should works correctly.