Bumping package.json version without invalidating docker cache

Here's my take on this, based off other answers, but shorter and with usage of jq:

Dockerfile:

FROM endeveit/docker-jq AS deps

# https://stackoverflow.com/a/58487433
# To prevent cache invalidation from changes in fields other than dependencies

COPY package.json /tmp

RUN jq '{ dependencies, devDependencies }' < /tmp/package.json > /tmp/deps.json

FROM node:12-alpine

WORKDIR /app

COPY --from=deps /tmp/deps.json ./package.json
COPY package-lock.json .

RUN npm ci
# https://docs.npmjs.com/cli/ci.html#description

COPY . .

RUN npm run build

LABEL maintainer="Alexey Vishnyakov <[email protected]>"

I extract dependencies and devDependencies fields to a separate file, then on next build step I copy it from the previous step as package.json (COPY --from=deps /tmp/deps.json ./package.json).

After RUN npm ci, COPY . . will overwrite gutted package.json with the original one (you can test it by adding RUN cat package.json after COPY . . command.

Note that npm-scripts commands like postinstall won't run since they're not present in the file during npm ci and also if npm ci is running from root and without --unsafe-perm

Either run commands after COPY . . or/and (if needed) include them via jq (changing command will invalidate cache layer) or add --unsafe-perm

Dockerfile:

FROM endeveit/docker-jq AS deps

COPY package.json /tmp

RUN jq '{ dependencies, devDependencies, peerDependencies, scripts: (.scripts | { postinstall }) }' < /tmp/package.json > /tmp/deps.json
# keep postinstall script 

FROM node:12-alpine

WORKDIR /app

COPY --from=deps /tmp/deps.json ./package.json
COPY package-lock.json .

# RUN npm ci --unsafe-perm 
# allow postinstall to run from root (security risk)

RUN npm ci
# https://docs.npmjs.com/cli/ci.html#description

RUN npm run postinstall

...

I spent some time thinking about this. Fundamentally, I'm cheating because the package.json file is, in fact, changed, which means anything that circumvents the cache invalidation technically makes the build not reproducible.

For my purposes, however, I care more about build time than strict cache correctness. Here's what I came up with:

build-artifacts.js

/*
Used to keep docker cache fresh despite package.json version bumps.

In this script
- copy package.json to package-artifact.json
- zero package.json version

In Docker
- copy package.json
- run npm install normal
- copy package-artifact.json to package.json (undo-build-artifacts.js accomplishes this with a conditional check that package-artifact exists)
*/

const fs = require('fs');
const package = fs.readFileSync('package.json', 'utf8');
fs.writeFileSync('package-artifact.json', package);
const modifiedPackage = { ...JSON.parse(package), version: '0.0.0' };
fs.writeFileSync('package.json', JSON.stringify(modifiedPackage));

const packageLock = fs.readFileSync('package-lock.json', 'utf8');
fs.writeFileSync('package-lock-artifact.json', packageLock);
const modifiedPackageLock = { ...JSON.parse(packageLock), version: '0.0.0' };
fs.writeFileSync('package-lock.json', JSON.stringify(modifiedPackageLock));

undo-build-artifacts.js

const fs = require('fs');

const hasBuildArtifacts = fs.existsSync('package-artifact.json');
if (hasBuildArtifacts) {
  const package = fs.readFileSync('package-artifact.json', 'utf8');
  const packageLock = fs.readFileSync('package-lock-artifact.json', 'utf8');

  fs.writeFileSync('package.json', package);
  fs.writeFileSync('package-lock.json', packageLock);

  fs.unlinkSync('package-artifact.json');
  fs.unlinkSync('package-lock-artifact.json');
}

These two files serve to relocate package.json and package-lock.json, replacing them with artifacts that have zeroed-out versions. These artifacts will be used in the docker build, and will be replaced with the original versions upon npm install completion.

I run build-artifacts.js in a Travis CI before_script, and undo-build-artifacts.js in the Dockerfile itself (after I npm install). undo-build-artifacts.js incorporates a check for the build artifacts, meaning the Docker container can still build if build-artifacts.js hasn't run. That keeps the container portable enough in my books. :)

You can add an additional "preparation" step in your Dockerfile that creates a temporary package.json where the "version" field is fixed. This file is then used while installing dependencies and afterwards replaced by the "real" package.json.
As all of this happens during the Docker build process, your actual source repository is not touched (so you can use the environment variable npm_package_version both during your build and when running the docker script, e.g. to tag) and the solution is portable:

Dockerfile:

# PREPARATION
FROM node:lts-alpine as preparation
COPY package.json package-lock.json ./
# Create temporary package.json where version is set to 0.0.0
# – this way the cache of the build step won't be invalidated
# if only the version changed.
RUN ["node", "-e", "\
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf-8'));\
const pkgLock = JSON.parse(fs.readFileSync('package-lock.json', 'utf-8'));\
fs.writeFileSync('package.json', JSON.stringify({ ...pkg, version: '0.0.0' }));\
fs.writeFileSync('package-lock.json', JSON.stringify({ ...pkgLock, version: '0.0.0' }));\
"]

# BUILD
FROM node:lts-alpine as build
# Install deps, using temporary package.json from preparation step
COPY --from=preparation package.json package-lock.json ./
RUN npm ci
# Copy source files (including "real" package.json) and build app
COPY . .
RUN npm run build

If you think inlining the Node script is iffy (I like it, because this way the entire Docker build process can be found in the Dockerfile), you can of course extract it to a separate JS file:

create-tmp-pkg.js:

const fs = require('fs');
const pkg = JSON.parse(fs.readFileSync('package.json', 'utf-8'));
const pkgLock = JSON.parse(fs.readFileSync('package-lock.json', 'utf-8'));

fs.writeFileSync('package.json', JSON.stringify({ ...pkg, version: '0.0.0' }));
fs.writeFileSync('package-lock.json', JSON.stringify({ ...pkgLock, version: '0.0.0' }));

and change your preparation step to:

# PREPARATION
FROM node:lts-alpine as preparation
COPY package.json package-lock.json create-tmp-pkg.js ./
# Create temporary package.json where version is set to "0.0.0"
# – this way the cache of the build step won't be invalidated
# if only the version changed.
RUN node create-tmp-pkg.js

I went about this a bit different. I just ignore the version in package.json and leave it set to 1.0.0. Instead I add a file version.json then I use a script like the one below for deploying.

This approach won't work if you need to publish to npm, since the version will never change

version.json

{"version":"1.2.3"}

deploy.sh

#!/bin/sh
VERSION=`node -p "require('./version.json').version"`

#docker build
docker pull node:10
docker build . -t mycompany/myapp:v$VERSION

#commit version tag
git add version.json
git commit  -m "version $VERSION"
git tag v$VERSION
git push origin
git push origin v$VERSION

#push Docker image to repo
docker push mycompany/myapp:v$VERSION

I normally just update the version file manually but if you want something that works like npm version you can use a script like this that uses the semvar package.

patch.js

var semver = require('semver')
var fs = require('fs')
var version = require('./version.json').version
var patch = semver.inc(version, 'patch')

fs.writeFile('./version.json', JSON.stringify({'version': patch}), (err) => {
  if (err) {
    console.error(err)
  } else {
    console.log(version + ' -> ' + patch)
  }
})