Hosting docker images on multiple registries but reference them the same way e.g. in a k8s deployment

Solution 1:

There are some workarounds to achieve the result like you've described in your question:

• Helm Chart
• Kustomize
• Registry

Helm Chart

As you've said in the question:

The only other solution I can think of, is to build everything into helm charts, make the first part of the image reference a variable and pass the registry as a variable to the chart.

This is subjectively most convenient way to specify the repositories for Deployment.

The example of it could like below:

deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu
spec:
  selector:
    matchLabels:
      app: ubuntu
  replicas: 1
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      containers:
      - name: "{{ .Values.image.repo }}{{ .Values.image.name }}:{{ .Values.image.tag }}"  
        image: 
        command:
        - sleep 
        - "infinity"

Values inside of a template could be provided either by:

• values.yaml file:

image: 
  repo: gcr.io/ # <-- only an example
  name: ubuntu
  tag: latest

• $ helm command like:

$ helm install ubuntu . --set=image.repo=eu.gcr.io/

A tip!

Specifying --set parameter with already existing value in values.yaml will override it (in this example)!

Kustomize

You can also use kustomize to "render" the Deployment with their respective repositories and images.

There is quite extensive explanation on how it can be done in below blog post:

• Blog.stack-labs.com: Code: Kustomize 101

Example of such solution (by following above link) could be boiled down to:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  selector:
    matchLabels:
      app: sl-demo-app
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - name: app
        image: foo/bar:latest
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

custom-image.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  template:
    spec:
      containers:
        - name: app # IMPORTANT 
          image: eu.gcr.io/ubuntu:latest

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

patchesStrategicMerge:
- custom-image.yaml

After running above example with:

• kustomize build

You should see changes in image:

• from: image: foo/bar:latest
• to: image: eu.gcr.io/ubuntu:latest

You could also take a look on this StackOverflow answer:

• Stackoverflow.com: Questions: Kustomize add imagepullsecrets to all deployments

Registry

You can also take a look on registry project such as Harbor.

You can read on their github page:

Harbor is an open source trusted cloud native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity and management.

Features

• Cloud native registry: With support for both container images and Helm charts, Harbor serves as registry for cloud native environments like container runtimes and orchestration platforms.

• Role based access control: Users access different repositories through 'projects' and a user can have different permission for images or Helm charts under a project.

• Policy based replication: Images and charts can be replicated (synchronized) between multiple registry instances based on policies with using filters (repository, tag and label). Harbor automatically retries a replication if it encounters any errors. This can be used to assist loadbalancing, achieve high availabiliy, and faciliate multi-datacenter deployments in hybrid and multi-cloud scenarios.

...

-- Github.com: Goharbor: Harbor

With it you could:

• Create repositories that could be used only by authenticated and authorized accounts.
• Use it for replication purposes between locations.

Adding to that you could also look on Docker Registry proxy like the one below:

• Hub.docker.com: Rpardini: Docker registry proxy

Additional resources:

• Helm.sh
• Github.com: Kubernetes sigs: Kustomize