Unable to load kubernetes dashboard after successful oauth2
I have deployment my kubernetes cluster v1.23.1 with kubeadm and configured it with the keycloak identity provider for authentication.
API server configuration for keycloak IDP
...
- --oidc-issuer-url=https://kubemaster:8443/auth/realms/local
- --oidc-client-id=gatekeeper
- --oidc-username-claim=name
- --oidc-groups-claim=groups
- --oidc-ca-file=/etc/kubernetes/ssl/kubemaster.crt
...
I am running oauth2-proxy in another pod which authenticates with keycloak idp and provides the tokens (id_token). Here is my oauth2-proxy deployment with the service to expose it.
---
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: oauth2-proxy
name: oauth2-proxy
namespace: kubernetes-dashboard
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: oauth2-proxy
template:
metadata:
labels:
k8s-app: oauth2-proxy
spec:
containers:
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:latest
imagePullPolicy: Always
ports:
- containerPort: 4180
protocol: TCP
args:
- --provider=oidc
- --email-domain=*
- --http-address=0.0.0.0:4180
env:
- name: OAUTH2_PROXY_OIDC_ISSUER_URL
value: https://192.168.122.54:8443/auth/realms/local
- name: OAUTH2_PROXY_REDIRECT_URL
value: https://kubernetes-dashboard.localdev.me:8081/oauth2/callback
- name: OAUTH2_PROXY_CLIENT_ID
value: gatekeeper
- name: OAUTH2_PROXY_CLIENT_SECRET
value: jZzvJ0wCDDwltV3tAf0SXSbVoKXM1RqV
- name: OAUTH2_PROXY_COOKIE_SECRET
value: kgKUT3IMmESA81VWXvRpYIYwMSo1xndwIogUks6IS00=
- name: OAUTH2_PROXY_UPSTREAM
value: https://kubernetes-dashboard
- name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY
value: "true"
- name: OAUTH2_PROXY_COOKIE_DOMAIN
value: .localdev.me
- name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
value: "true"
- name: OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER
value: "true"
- name: OAUTH2_PROXY_SSL_UPSTREAM_INSECURE_SKIP_VERIFY
value: "true"
- name: OAUTH2_PROXY_OIDC_EMAIL_CLAIM
value: email
- name: OAUTH2_PROXY_GROUPS_CLAIM
value: groups
- name: OAUTH2_PROXY_ALLOWED_GROUPS
value: developers
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: oauth2-proxy
name: oauth2-proxy
namespace: kubernetes-dashboard
spec:
#type: NodePort
ports:
- port: 4180
targetPort: 4180
name: http
selector:
k8s-app: oauth2-proxy
I have nginx ingress deployed in front to route the request to oauth2-proxy as shown below.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: oauth-proxy
namespace: kubernetes-dashboard
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: kubernetes-dashboard.localdev.me
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: oauth2-proxy
port:
number: 4180
tls:
- hosts:
- kubernetes-dashboard.localdev.me
secretName: kubernetes-dashboard-ingress-tls
I am using port forward from localhost to ingress resource on port 443.
kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8081:443
The issue currently i am facing is i am able to launch the url - https://kubernetes-dashboard.localdev.me:8081/ which routes to oauth2 proxy page from there i am able to launch the keycloak realm login page which i can authenticate. But once the keycloak authentication is successfully done i am not able to load the upstream which is kubernetes-dashboard service with the id_token. Instead i am getting a 404 not found for https://kubernetes-dashboard.localdev.me:8081/
oauth2-proxy Log details
[2022/01/04 19:06:24] [oauthproxy.go:866] No valid authentication in request. Initiating login.
192.168.1.169:33826 - 48490760d6e0fc519e13d3158878112a - - [2022/01/04 19:06:24] kubernetes-dashboard.localdev.me:8081 GET - "/" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 403 8033 0.000
[2022/01/04 19:06:25] [oauthproxy.go:866] No valid authentication in request. Initiating login.
192.168.1.169:33826 - a87fbdc228026d99e6fc9f29d9ffc6df - - [2022/01/04 19:06:25] kubernetes-dashboard.localdev.me:8081 GET - "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 403 8044 0.000
192.168.1.169:33826 - 3c052e49f9f77e89c561f3d0bef47fed - - [2022/01/04 19:06:43] kubernetes-dashboard.localdev.me:8081 GET - "/oauth2/start?rd=%2F" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 302 356 0.000
192.168.1.169:33826 - a16b9b1b22ec03e58b9b930caa325f5b - [email protected] [2022/01/04 19:06:55] [AuthSuccess] Authenticated via OAuth2: Session{email:[email protected] user:52ed0190-4d9c-4749-90f4-37e5896bdc42 PreferredUsername:alice token:true id_token:true created:2022-01-04 19:06:55.77587057 +0000 UTC m=+290.895007232 expires:2022-01-04 19:11:55.775499536 +0000 UTC m=+590.894636200 refresh_token:true groups:[developers]}
192.168.1.169:33826 - a16b9b1b22ec03e58b9b930caa325f5b - - [2022/01/04 19:06:55] kubernetes-dashboard.localdev.me:8081 GET - "/oauth2/callback?state=HBDoNuX3mudlwQrjdEkxYP9yjgHn5_mqOXk8T-G21dg%3A%2F&session_state=e928739a-c94d-49ba-80a9-a1f83a936bf1&code=9a592559-236c-4896-86b4-ba8f657821e6.e928739a-c94d-49ba-80a9-a1f83a936bf1.b7b62a18-269b-480a-b5bd-ad3c16f94394" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 302 24 0.030
192.168.1.169:33826 - 9288833c2f8e82f2ad94c6260bcf68c7 - [email protected] [2022/01/04 19:06:55] kubernetes-dashboard.localdev.me:8081 GET - "/" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 404 19 0.000
Ingress controller logs
127.0.0.1 - - [04/Jan/2022:19:02:26 +0000] "GET / HTTP/2.0" 404 19 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 2842 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 19 0.000 404 d19c25394498799f5024567ddd1fafcd
127.0.0.1 - - [04/Jan/2022:19:06:24 +0000] "GET / HTTP/2.0" 403 8033 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 297 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 8060 0.000 403 48490760d6e0fc519e13d3158878112a
127.0.0.1 - - [04/Jan/2022:19:06:25 +0000] "GET /favicon.ico HTTP/2.0" 403 8044 "https://kubernetes-dashboard.localdev.me:8081/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 102 0.003 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 8071 0.004 403 a87fbdc228026d99e6fc9f29d9ffc6df
127.0.0.1 - - [04/Jan/2022:19:06:43 +0000] "GET /oauth2/start?rd=%2F HTTP/2.0" 302 356 "https://kubernetes-dashboard.localdev.me:8081/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 32 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 356 0.000 302 3c052e49f9f77e89c561f3d0bef47fed
127.0.0.1 - - [04/Jan/2022:19:06:55 +0000] "GET /oauth2/callback?state=HBDoNuX3mudlwQrjdEkxYP9yjgHn5_mqOXk8T-G21dg%3A%2F&session_state=e928739a-c94d-49ba-80a9-a1f83a936bf1&code=9a592559-236c-4896-86b4-ba8f657821e6.e928739a-c94d-49ba-80a9-a1f83a936bf1.b7b62a18-269b-480a-b5bd-ad3c16f94394 HTTP/2.0" 302 24 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 351 0.031 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 24 0.028 302 a16b9b1b22ec03e58b9b930caa325f5b
127.0.0.1 - - [04/Jan/2022:19:06:55 +0000] "GET / HTTP/2.0" 404 19 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 2827 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 19 0.004 404 9288833c2f8e82f2ad94c6260bcf68c7
Please suggest on how i can route the authenticated session with id_token to kubernetes dashboard service for loading the dashboard.
Solution 1:
I was finally able to resolve my issue with some updates to my yaml definition files.
Assuming you have a kubernetes cluster installed v1.23.1 with kubeadm on Ubuntu 20.04 and setup networking with flannel networking --pod-network-cidr=10.244.0.0/16. Also you have the keycloak oidc service setup (image - quay.io/keycloak/keycloak:16.1.0). Here are the updated yml definition files which helped me to resolve this issue.
Ingress controller applied -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/cloud/deploy.yaml
kubernetes dashboard applied -
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.4.0/aio/deploy/recommended.yaml
oauth2-proxy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: oauth2-proxy
name: oauth2-proxy
namespace: kubernetes-dashboard
spec:
replicas: 1
selector:
matchLabels:
k8s-app: oauth2-proxy
template:
metadata:
labels:
k8s-app: oauth2-proxy
spec:
containers:
- args:
- --provider=oidc
- --email-domain=*
- --http-address=0.0.0.0:4180
env:
- name: OAUTH2_PROXY_OIDC_ISSUER_URL
value: https://kubemaster:8443/auth/realms/local
- name: OAUTH2_PROXY_REDIRECT_URL
value: https://<FQDN>/oauth2/callback
- name: OAUTH2_PROXY_CLIENT_ID
value: gatekeeper
- name: OAUTH2_PROXY_CLIENT_SECRET
value: jZzvJ0wCDDwltV3tAf0SXSbVoKXM1RqV
- name: OAUTH2_PROXY_COOKIE_SECRET
value: kgKUT3IMmESA81VWXvRpYIYwMSo1xndwIogUks6IS00=
- name: OAUTH2_PROXY_UPSTREAM
value: https://kubernetes-dashboard
- name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY
value: "true"
- name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
value: "true"
- name: OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER
value: "true"
- name: OAUTH2_PROXY_SSL_UPSTREAM_INSECURE_SKIP_VERIFY
value: "true"
- name: OAUTH2_PROXY_OIDC_EMAIL_CLAIM
value: email
- name: OAUTH2_PROXY_GROUPS_CLAIM
value: groups
- name: OAUTH2_PROXY_ALLOWED_GROUPS
value: developers
- name: OAUTH2_PROXY_SKIP_PROVIDER_BUTTON
value: "true"
- name: OAUTH2_PROXY_SET_AUTHORIZATION_HEADER
value: "true"
image: quay.io/oauth2-proxy/oauth2-proxy:latest
imagePullPolicy: Always
name: oauth2-proxy
ports:
- containerPort: 4180
protocol: TCP
dashboard-ingress.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/auth-response-headers: "authorization"
name: external-auth-oauth2
namespace: kubernetes-dashboard
spec:
ingressClassName: nginx
rules:
- host: __INGRESS_HOST__
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kubernetes-dashboard
port:
number: 443
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: oauth2-proxy
namespace: kubernetes-dashboard
spec:
ingressClassName: nginx
rules:
- host: __INGRESS_HOST__
http:
paths:
- path: /oauth2
pathType: Prefix
backend:
service:
name: oauth2-proxy
port:
number: 4180
tls:
- hosts:
- __INGRESS_HOST__
secretName: __INGRESS_SECRET__
The issue with keycloak oidc provider its unable to set the Authroization header for which we need to manaully add the set authorization header in the oauth2-proxy defintion and also this header need to be appended in the response from the ingress to the upstream kubernetes dashboard by setting annotation - nginx.ingress.kubernetes.io/auth-response-headers: "authorization"
Also adding a reference for kubernetes dashboard authentication using github oauth2 which i was following to resolve this issue.
https://kubernetes.github.io/ingress-nginx/examples/auth/oauth-external-auth/#example-oauth2-proxy-kubernetes-dashboard