LXD: Running an image with a foreign architecture

I want to use LXD to bootstrap a root filesystem for deployment on an embedded ARM system from my AMD64 host running ubuntu 16.04. Previously, I've done this with a script and the chroot command, but the script I was using was error-prone and had a bad habit of deleting my /dev entries.

I have been able to copy the image locally using lxc image copy images:ubuntu/16.04/arm64 --alias=ubuntu-server-arm64, and have installed qemu-user-static, but am not able to actually launch a container from this image. I get the following error:

$ lxc launch ubuntu-server-arm64 bootstrap
Creating bootstrap
error: Requested architecture isn't supported by this host

Is there a way to force lxd to ignore the architecture mismatch and use qemu-user-static to run the child container?

I was able to get a foreign architecture to work with LXD. I took an ARMv7 image, created a squashfs from it, added qemu-arm-static under /usr/bin. The trick to get it to work with LXD was to have the meta-data tarball say the image is x86_64 (or whatever your host architecture is). Importing and running this image works, LXD thinks it's a supported architecture (matches host) and its able to run because of qemu-arm-static and binfmt.

ALPINE_MIRROR_URL="http://ams.edge.kernel.org/alpine"
ALPINE_VERSION="3.11.5"
ALPINE_ARCH="armhf"

version=$(echo $ALPINE_VERSION | cut -d '.' -f 1-2)
curl -O ${ALPINE_MIRROR_URL}/v${version}/releases/${ALPINE_ARCH}/alpine-minirootfs-${ALPINE_VERSION}-${ALPINE_ARCH}.tar.gz

ROOTFS=alpine-rootfs
mkdir ${ROOTFS}
tar xzf alpine-minirootfs-${ALPINE_VERSION}-${ALPINE_ARCH}.tar.gz -C ${ROOTFS}

cp /usr/bin/qemu-arm-static ${ROOTFS}/usr/bin/
mksquashfs ${ROOTFS} rootfs.squashfs

lxc import meta.tar.xz rootfs.squashfs --alias crossx
lxc launch crossx x
lxc exec x -- uname -a
Linux x 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 armv7l Linux
lxc image ls
| crossx   | 1e5e5a2dafb8 | no     | Alpinelinux  x86_64 (20200330_1354) | x86_64       | CONTAINER | 5.45MB | Apr 2, 2020 at 6:25pm (UTC)  |

You can see in the last two commands the container is really armv7l but LXD thinks its x86_64.

According to Stephane Graber who is in charge of the LXC/LXD project (in response to an inquiry from me on their discussions site):

qemu-user-static is a binfmt helper which lets you do on the fly conversion between architectures. It effectively lets you run binaries for architectures other than your current one.

qemu-user-static itself should work fine inside a container and will let you run some binaries of foreign architectures inside it.

Trying to run an entire container through qemu-user-static is very impractical due to some big limitations of qemu-user-static, for example, anything that relies on ptrace (init systems and debugging tools), netlink (all the network tools and some init systems) or thread (a lot more software) will usually fail miserably.

The approach I implemented back in LXC was to create a mixed container where most packages were of the foreign architecture but the init system, network tools, … were of the native architecture. This was kinda-working but also not particularly useful, not to mention very slow.

As this is effectively unsupportable as far as we’re concerned, we are not providing any foreign-architecture images for LXD. You could however build your own by assembling a rootfs of a foreign architecture, then including the needed qemu-user-static binaries, replacing any binary that will not work with emulation and generate this as a LXD image (marking it with the architecture it’s intended to run on rather than the architecture it contains).

So, effectively, attempting to run a foreign architecture on LXD is unsupported, and the LXC method was more or less a 'mixed environment' type of hack.

It sounds like it could also be done, but would require you to create a rootfs of a foreign architecture and basically create the 'image' yourself that runs all this, which for various reasons could be a headache and a lot of work to accomplish.