pushing an image with two tags to gcr.io results in two different images

I'm doing the following:

docker build -t gcr.io/projid/imgname:333 -t gcr.io/projid/imgname:latest .

docker login -u _json_key -p "$(cat /secrets/service-account.json)" https://gcr.io

docker push gcr.io/projid/imgname:333

docker push gcr.io/projid/imgname:latest

Output of 1st push command:

Pushing to google container registry ...
The push refers to a repository [gcr.io/projid/imgname]
24af4f7c7118: Preparing
17b0972980d8: Preparing
6d6a6425aacb: Preparing
809c8c0dd73c: Preparing
e8d45b8ab3ca: Preparing
e8fa134cb7b8: Preparing
7cbcbac42c44: Preparing
e8fa134cb7b8: Waiting
7cbcbac42c44: Waiting
809c8c0dd73c: Layer already exists
e8d45b8ab3ca: Layer already exists
7cbcbac42c44: Layer already exists
e8fa134cb7b8: Layer already exists
17b0972980d8: Pushed
24af4f7c7118: Pushed
6d6a6425aacb: Pushed
333: digest: sha256:dae8cf914ba49928e6f0a34f6740802403813e6d10aa1c1d448a62ce9bb69066 size: 1779

Output of 2nd push command:

Pushing to google container registry ...
The push refers to a repository [gcr.io/projid/imgname]
24af4f7c7118: Preparing
17b0972980d8: Preparing
6d6a6425aacb: Preparing
809c8c0dd73c: Preparing
e8d45b8ab3ca: Preparing
e8fa134cb7b8: Preparing
7cbcbac42c44: Preparing
e8fa134cb7b8: Waiting
7cbcbac42c44: Waiting
e8d45b8ab3ca: Layer already exists
809c8c0dd73c: Layer already exists
24af4f7c7118: Layer already exists
17b0972980d8: Layer already exists
6d6a6425aacb: Layer already exists
7cbcbac42c44: Layer already exists
e8fa134cb7b8: Layer already exists
latest: digest: sha256:4f57375919829982741d095f8917306fe0c1410e115d030179bae4b8e4299c30 size: 1742

Question: Why the same image with two tags results in two different images in google container registry?

Solution 1:

You are certainly pushing two image tags. I suggest building with a single tag, then adding the second one and pushing the image tags one by one. If the second one adds a new digest is a GCR bug. I have ran into this bug and solved by deleting the repository so it gets re-created at the next push. In my configuration the Docker version was also a factor. Version 17.04.0-ce, build 4845c56 would trigger the extra digest while version 17.03.1-ce, build c6d412e would work fine.

Solution 2:

UPDATE: May 2021

I don't receive this issue, so I believe it's fixed:

Already have image (with digest): gcr.io/cloud-builders/docker
The push refers to repository [gcr.io/proj-dev/client]
1b046f5a4242: Preparing
45ad1ef5106f: Preparing
02c95cff5c48: Preparing
4d5085b7c406: Preparing
516e4ad96ca6: Preparing
94ed6f39e7b4: Preparing
e73bffd94869: Preparing
ebf12965380b: Preparing
94ed6f39e7b4: Waiting
e73bffd94869: Waiting
ebf12965380b: Waiting
516e4ad96ca6: Pushed
4d5085b7c406: Pushed
e73bffd94869: Layer already exists
ebf12965380b: Layer already exists
1b046f5a4242: Pushed
45ad1ef5106f: Pushed
94ed6f39e7b4: Pushed
02c95cff5c48: Pushed
latest: digest: sha256:f824ee7aecb79e1826a311d4dd0d85919a636abc7defc165ec36e8ceb330829c size: 1999
The push refers to repository [gcr.io/proj-dev/client]
1b046f5a4242: Preparing
45ad1ef5106f: Preparing
02c95cff5c48: Preparing
4d5085b7c406: Preparing
516e4ad96ca6: Preparing
94ed6f39e7b4: Preparing
e73bffd94869: Preparing
ebf12965380b: Preparing
94ed6f39e7b4: Waiting
e73bffd94869: Waiting
ebf12965380b: Waiting
1b046f5a4242: Layer already exists
02c95cff5c48: Layer already exists
4d5085b7c406: Layer already exists
45ad1ef5106f: Layer already exists
516e4ad96ca6: Layer already exists
e73bffd94869: Layer already exists
ebf12965380b: Layer already exists
94ed6f39e7b4: Layer already exists
3.0.0-master-7e0d787: digest: sha256:f824ee7aecb79e1826a311d4dd0d85919a636abc7defc165ec36e8ceb330829c size: 1999

Solution 3:

With saying two different images are you referring to two different digest values?

SHA stands for Secure Hash Algorithm and it is a family of cryptographic hash functions and SHA-256 algorithm generates an almost-unique, fixed size 256-bit (32-byte) hash. Since the image tag is also being used to calculate the digest, the value of digest will be changed significantly even only one bit of the tag changes. In cryptography, this called avalanche effect and you can read more about it in here.

Solution 4:

One thing to check is to compare the manifests of the two images. For example:

docker manifest inspect <image>

or

skopeo inspect --raw docker://image

If you diff the JSON output of those commands between the two images that should be the same but are not, it might give you some insight into the underlying problem.

For example, in my case, it turns out that it was a bug in my image build tool (buildah, via podman), in which two of the image layers had an incorrect media type of application/vnd.docker.image.rootfs.diff.tar.gzip, when in fact the correct media type was application/vnd.docker.image.rootfs.diff.tar.

Furthermore, podman was (helpfully?) correcting the media type when it pushed the same image with a different tag, so the container registry was actually ending up with the tags on two different "images".