Kafka Broker Runs locally via docker compose but fails in kubernetes with near identical config

Kafka broker runs beautifully when running locally via docker-compose.

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

When running via kubernetes though

apiVersion: v1
kind: Service
metadata:
  name: zevrant-kafka-zookeeper-service
spec:
  ports:
    - port: 443
      targetPort: 2181
  selector:
    app: zevrant-kafka-zookeeper-service
---
apiVersion: v1
kind: Service
metadata:
  name: zevrant-kafka-service
spec:
  type: NodePort
  ports:Connection to node -1 (/<IP_ADDRESS>:30129) could not be established. Broker may not be available.
    - port: 443
      targetPort: 9092
      nodePort: 30129
  selector:
    app: zevrant-kafka-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zevrant-kafka-zookeeper-service-deployment
  labels:
    app: zevrant-kafka-zookeeper-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zevrant-kafka-zookeeper-service
  template:
    metadata:
      labels:
        app: zevrant-kafka-zookeeper-service
    spec:
      volumes:
        - name: zookeeper-data
          nfs:
            server: <IP_ADDRESS>
            path: /zookeeper
      imagePullSecrets:
        - name: regcred
      nodeSelector:
        architecture: amd64
      containers:
        - name: zevrant-kafka-zookeeper-service
          image: bitnami/zookeeper:3.7.0
          env:
            - name: ALLOW_ANONYMOUS_LOGIN
              value: "yes"
          ports:
            - containerPort: 2181
          volumeMounts:
            - name: zookeeper-data
              mountPath: /bitnami/zookeeper
        - name: zevrant-kafka-zookeper-ui-service
          image: elkozmon/zoonavigator:1.1.0
          ports:
            - containerPort: 9000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zevrant-kafka-service-daemonset
  labels:
    app: zevrant-kafka-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zevrant-kafka-service
  template:
    metadata:
      labels:
        app: zevrant-kafka-service
    spec:
      volumes:
        - name: kafka-storage
          hostPath:
            path: /opt/kafka/dev/
      imagePullSecrets:
        - name: regcred
      nodeSelector:
        architecture: amd64
      containers:
        - name: zevrant-kafka-broker-service
          image: bitnami/kafka:2.8.0
          volumeMounts:
            - name: kafka-storage
              mountPath: /bitnami/kafka
          ports:
            - containerPort: 9092
          env:
            - name: ALLOW_PLAINTEXT_LISTENER
              value: "yes"
            - name: KAFKA_CFG_ZOOKEEPER_CONNECT
              value: "zevrant-kafka-zookeeper-service:443"
            - name: KAFKA_BROKER_ID
              value: "1"
            - name: KAFKA_CFG_LISTENERS
              value: "PLAINTEXT://0.0.0.0:9092"
            - name: KAFKA_CFG_ADVERTISED_LISTENERS
              value: "PLAINTEXT://<IP_ADDRESS>:30129"

The broker starts but repeatedly spams the error

INFO [Admin Manager on Broker 1]: Error processing create topic request CreatableTopic(name='zevrant-video-stream', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)

I used the same command on both clusters to create the topic aside from the connection details and in both cases I ran the command from inside the running broker container.

Locally

kafka-topics.sh --create --topic zevrant-video-stream --zookeeper localhost:2181 --partitions 1 --replication-factor 1 --config retention.ms=86400000

Kubernetes

kafka-topics.sh --create --topic zevrant-video-stream --zookeeper zevrant-kafka-zookeeper-service:443 --partitions 1 --replication-factor 1 --config retention.ms=86400000

I'm not really sure why this works in one place and not the other.

Attempting to connect a producer to the kubernetes broker i receive

Connection to node -1 (/<IP_ADDRESS>:30129) could not be established. Broker may not be available.

When querying zookeeper for broker information, zookeeper shows the broker as connected

zkCli.sh get /brokers/ids/1


{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://<IP_ADDRESS>:9092"],"jmx_port":-1,"features":{},"host":"<IP_ADDRESS>","timestamp":"1622398259745","port":9092,"version":5}

What would cause the broker to connected, online, but unavailable?

Solution 1:

turns out there is additional configuration needed for running inside a closed network as discussed here