Integrating Python Poetry with Docker

Can you give me an example of a Dockerfile in which I can install all the packages I need from poetry.lock and pyproject.toml into my image/container from Docker?

There are several things to keep in mind when using poetry together with docker.

Installation

Official way to install poetry is via:

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

This way allows poetry and its dependencies to be isolated from your dependencies. But, in my point of view, it is not a very good thing for two reasons:

1. poetry version might get an update and it will break your build. In this case you can specify POETRY_VERSION environment variable. Installer will respect it
2. I do not like the idea to pipe things from the internet into my containers without any protection from possible file modifications

So, I use pip install 'poetry==$POETRY_VERSION'. As you can see, I still recommend to pin your version.

Also, pin this version in your pyproject.toml as well:

[build-system]
# Should be the same as `$POETRY_VERSION`:
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"

It will protect you from version mismatch between your local and docker environments.

Caching dependencies

We want to cache our requirements and only reinstall them when pyproject.toml or poetry.lock files change. Otherwise builds will be slow. To achieve working cache layer we should put:

COPY poetry.lock pyproject.toml /code/

After the poetry is installed, but before any other files are added.

Virtualenv

The next thing to keep in mind is virtualenv creation. We do not need it in docker. It is already isolated. So, we use poetry config virtualenvs.create false setting to turn it off.

Development vs Production

If you use the same Dockerfile for both development and production as I do, you will need to install different sets of dependencies based on some environment variable:

poetry install $(test "$YOUR_ENV" == production && echo "--no-dev")

This way $YOUR_ENV will control which dependencies set will be installed: all (default) or production only with --no-dev flag.

You may also want to add some more options for better experience:

1. --no-interaction not to ask any interactive questions
2. --no-ansi flag to make your output more log friendly

Result

You will end up with something similar to:

FROM python:3.6.6-alpine3.7

ARG YOUR_ENV

ENV YOUR_ENV=${YOUR_ENV} \
  PYTHONFAULTHANDLER=1 \
  PYTHONUNBUFFERED=1 \
  PYTHONHASHSEED=random \
  PIP_NO_CACHE_DIR=off \
  PIP_DISABLE_PIP_VERSION_CHECK=on \
  PIP_DEFAULT_TIMEOUT=100 \
  POETRY_VERSION=1.0.0

# System deps:
RUN pip install "poetry==$POETRY_VERSION"

# Copy only requirements to cache them in docker layer
WORKDIR /code
COPY poetry.lock pyproject.toml /code/

# Project initialization:
RUN poetry config virtualenvs.create false \
  && poetry install $(test "$YOUR_ENV" == production && echo "--no-dev") --no-interaction --no-ansi

# Creating folders, and files for a project:
COPY . /code

You can find a fully working real-life example here: wemake-django-template

Update on 2019-12-17

• Update poetry to 1.0

Multi-stage Docker build with Poetry and venv

Do not disable virtualenv creation. Virtualenvs serve a purpose in Docker builds, because they provide an elegant way to leverage multi-stage builds. In a nutshell, your build stage installs everything into the virtualenv, and the final stage just copies the virtualenv over into a small image.

Use poetry export and install your pinned requirements first, before copying your code. This will allow you to use the Docker build cache, and never reinstall dependencies just because you changed a line in your code.

Do not use poetry install to install your code, because it will perform an editable install. Instead, use poetry build to build a wheel, and then pip-install that into your virtualenv. (Thanks to PEP 517, this whole process could also be performed with a simple pip install ., but due to build isolation you would end up installing another copy of Poetry.)

Here's an example Dockerfile installing a Flask app into an Alpine image, with a dependency on Postgres. This example uses an entrypoint script to activate the virtualenv. But generally, you should be fine without an entrypoint script because you can simply reference the Python binary at /venv/bin/python in your CMD instruction.

Dockerfile

FROM python:3.7.6-alpine3.11 as base

ENV PYTHONFAULTHANDLER=1 \
    PYTHONHASHSEED=random \
    PYTHONUNBUFFERED=1

WORKDIR /app

FROM base as builder

ENV PIP_DEFAULT_TIMEOUT=100 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PIP_NO_CACHE_DIR=1 \
    POETRY_VERSION=1.0.5

RUN apk add --no-cache gcc libffi-dev musl-dev postgresql-dev
RUN pip install "poetry==$POETRY_VERSION"
RUN python -m venv /venv

COPY pyproject.toml poetry.lock ./
RUN poetry export -f requirements.txt | /venv/bin/pip install -r /dev/stdin

COPY . .
RUN poetry build && /venv/bin/pip install dist/*.whl

FROM base as final

RUN apk add --no-cache libffi libpq
COPY --from=builder /venv /venv
COPY docker-entrypoint.sh wsgi.py ./
CMD ["./docker-entrypoint.sh"]

docker-entrypoint.sh

#!/bin/sh

set -e

. /venv/bin/activate

while ! flask db upgrade
do
     echo "Retry..."
     sleep 1
done

exec gunicorn --bind 0.0.0.0:5000 --forwarded-allow-ips='*' wsgi:app

wsgi.py

import your_app

app = your_app.create_app()

TL;DR

I have been able to set up poetry for a Django project using postgres. After doing some research, I ended up with the following Dockerfile:

FROM python:slim

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE 1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED 1

# Install and setup poetry
RUN pip install -U pip \
    && apt-get update \
    && apt install -y curl netcat \
    && curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
ENV PATH="${PATH}:/root/.poetry/bin"

WORKDIR /usr/src/app
COPY . .
RUN poetry config virtualenvs.create false \
  && poetry install --no-interaction --no-ansi

# run entrypoint.sh
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

This is the content of entrypoint.sh:

#!/bin/sh

if [ "$DATABASE" = "postgres" ]
then
    echo "Waiting for postgres..."

    while ! nc -z $SQL_HOST $SQL_PORT; do
      sleep 0.1
    done

    echo "PostgreSQL started"
fi

python manage.py migrate

exec "$@"

Detailed Explanation

Some points to notice:

• I have decide to use slim instead of alpine as tag for the python image because even though alpine images are supposed to reduce the size of Docker images and speed up the build, with Python, you can actually end up with a bit larger image and that takes a while to build (read this article for more info).

• Using this configuration builds containers faster than using the alpine image because I do not need to add some extra packages to install Python packages properly.

• I am installing poetry directly from the URL provided in the documentation. I am aware of the warnings provided by sobolevn. However, I consider that it is better in the long term to use the lates version of poetry by default than relying on an environment variable that I should update periodically.

• Updating the environment variable PATH is crucial. Otherwise, you will get an error saying that poetry was not found.

• Dependencies are installed directly in the python interpreter of the container. It does not create poetry to create a virtual environment before installing the dependencies.

In case you need the alpine version of this Dockerfile:

FROM python:alpine

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE 1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED 1

# Install dev dependencies
RUN apk update \
    && apk add curl postgresql-dev gcc python3-dev musl-dev openssl-dev libffi-dev

# Install poetry
RUN pip install -U pip \
    && curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
ENV PATH="${PATH}:/root/.poetry/bin"

WORKDIR /usr/src/app
COPY . .
RUN poetry config virtualenvs.create false \
  && poetry install --no-interaction --no-ansi

# run entrypoint.sh
ENTRYPOINT ["/usr/src/app/entrypoint.sh"]

Notice that the alpine version needs some dependencies postgresql-dev gcc python3-dev musl-dev openssl-dev libffi-dev to work properly.

That's minimal configuration that works for me:

FROM python:3.7

ENV PIP_DISABLE_PIP_VERSION_CHECK=on

RUN pip install poetry

WORKDIR /app
COPY poetry.lock pyproject.toml /app/

RUN poetry config virtualenvs.create false
RUN poetry install --no-interaction

COPY . /app

Note that it is not as safe as @sobolevn's configuration.

As a trivia I'll add that if editable installs will be possible for pyproject.toml projects, a line or two could be deleted:

FROM python:3.7

ENV PIP_DISABLE_PIP_VERSION_CHECK=on

WORKDIR /app
COPY poetry.lock pyproject.toml /app/

RUN pip install -e .

COPY . /app