I'm getting a "Failed to pull image" error on my deployment in minikube with a local registry, despite other deployments using the same image registry working as expected.

My workflow is to build the image from a Dockerfile called docker.collection as below.

$ minikube start --insecure-registry localhost:5000 --memory 4096
$ docker build . -f docker.collection -t localhost:5000/collection:latest -t localhost:5000/collection:dev
$ docker push localhost:5000/collection
$ cat deployment.yaml | kubectl apply -f -

The deployment is created as desired, but the pods don't start, and the minikube dashboard shows:

Failed to pull image "localhost:5000/collection:dev": rpc error: code = Unknown desc = Error response from daemon: manifest for localhost:5000/collection:dev not found

I have no reason to believe there was any problem in building or pushing the image, and on the host machine, I can pull successfully...

$ docker pull localhost:5000/collection:dev
dev: Pulling from collection
Digest: sha256:8becfdd412f0b86ece8335aa5ee1aede75992d17f88739a28da5939eab28fde5
Status: Image is up to date for localhost:5000/collection:dev

What should else I be checking? How can I get more detail on the failure?

Further debugging

I've also checked:

• the minikube node can reach localhost:5000:

$ minikube ssh
$ ping localhost:5000
PING localhost:5000 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.126 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.041 ms
64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.058 ms
^C
--- localhost:5000 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.041/0.075/0.126 ms
$ 

• a minimal run of the image fails in the same way: kubectl run coll-test --image=localhost:5000/collection:dev yields the same events:

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  24s               default-scheduler  Successfully assigned dev/coll-test-cf4467b8d-8vpdk to minikube
  Normal   BackOff    23s               kubelet, minikube  Back-off pulling image "localhost:5000/collection:dev"
  Warning  Failed     23s               kubelet, minikube  Error: ImagePullBackOff
  Normal   Pulling    7s (x2 over 23s)  kubelet, minikube  Pulling image "localhost:5000/collection:dev"
  Warning  Failed     7s (x2 over 23s)  kubelet, minikube  Failed to pull image "localhost:5000/collection:dev": rpc error: code = Unknown desc = Error response from daemon: manifest for localhost:5000/collection:dev not found
  Warning  Failed     7s (x2 over 23s)  kubelet, minikube  Error: ErrImagePull

• the entries for the image exist in the repo:

$ curl http://localhost:5000/v2/_catalog
{"repositories":["collection","foo","bar"]}

Deployment and Dockerfile

# deployment.yaml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  namespace: dev
  name: collection
  labels:
    app: collection
spec:
  replicas: 1
  selector:
    matchLabels:
      app: collection
  template:
    metadata:
      labels:
        app: collection
    spec:
      containers:
      - name: collection
        image: "localhost:5000/collection:dev"
        command: ["bash", "./collection_entry.sh"]
        imagePullPolicy: Always

# docker.collection
FROM python:3

WORKDIR /code
COPY . /code/
RUN adduser --system --group --no-create-home celery
RUN mkdir /var/run/celery
RUN mkdir /var/log/celery
RUN chown celery:celery /var/run/celery
RUN chown celery:celery /var/log/celery
RUN chmod 755 /var/run/celery
RUN chmod 755 /var/log/celery

RUN pip3 -q install -r requirements.txt
USER celery

ENTRYPOINT ./collection_entry.sh

1) You have to run eval $(minikube docker-env)

2) Build the image with the Docker daemon of Minikube

docker build -t collection .

3) Set the image in the pod spec like the build tag - collection

4) Set the imagePullPolicy to Never, otherwise Kubernetes will try to download the image.

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  namespace: dev
  name: collection
  labels:
    app: collection
spec:
  replicas: 1
  selector:
    matchLabels:
      app: collection
  template:
    metadata:
      labels:
        app: collection
    spec:
      containers:
      - name: collection
        image: "collection"
        command: ["bash", "./collection_entry.sh"]
        imagePullPolicy: Never