Problem setting up a user-space LXC container

lxc

Several evenings, I already try to get an LXC installation working with user-space containers. Since it's going to be a new server, I don't care (yet) too much about distribution and release, so I tried Debian 7+8 and Ubuntu 14.04 and 15.04 as host and Ubuntu 15.04 and Debian 8 as guest. All of them have their problems. The summary:

Debian 7.8: lxc-create: This command has to be run as root (both guests)

Debian 8.0: lxc-create: Operation not permitted (both guests)

Ubuntu 14.04 w/ Ubuntu 15.04 guest: lxc-start: call to cgmanager_create_sync failed: invalid request / The container failed to start.

Ubuntu 14.04 w/ Debian 8 guest: lxc-create: ERROR: Couldn't find a matching image.

Ubuntu 15.04: lxc-start: call to cgmanager_move_pid_sync failed: invalid request / The container failed to start. (both guests)

I followed the tutorial https://help.ubuntu.com/lts/serverguide/lxc.html and https://linuxcontainers.org/lxc/getting-started/, and they don't look like it's so complicated.

I wrote a script to make the test reproducable (to be run as root on a freshly booted live cd). Can anyone tell me what's wrong with it?

#!/bin/sh


# need to be run as root

set -x

echo "==== SYSTEM INFO & INSTALL ===="

lsb_release -a

uname -a

apt-get update
apt-get install -y lxc
apt-get clean

lxc-checkconfig
ifconfig
brctl show

adduser testuser

cat /etc/subuid /etc/subgid

cat >/etc/lxc/lxc-usernet <<.e
# USERNAME TYPE BRIDGE COUNT
testuser veth lxcbr0 2
.e

sudo -u testuser -i mkdir -p .config/lxc

sudo -u testuser -i tee .config/lxc/default.conf <<.e
lxc.id_map = u 0 $(grep testuser /etc/subuid | cut -d: -f2) 65536
lxc.id_map = g 0 $(grep testuser /etc/subgid | cut -d: -f2) 65536

lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx
.e
cat /home/testuser/.config/lxc/default.conf

echo "==== TEST UBUNTU VIVID GUEST ===="

sudo -u testuser -i lxc-create -t download -n vivid1 -- -d ubuntu -r utopic -a amd64
#You just created an Ubuntu container (release=trusty, arch=amd64, variant=default)

#sudo -u testuser -i lxc-start -n vivid1 -d
#read press_enter_key_when_quit

if [ $? -eq 0 ]; then
  rm /tmp/log
  sudo -u testuser -i lxc-start -n vivid1 -l debug --logfile /tmp/log
  cat /tmp/log
fi

echo "==== TEST DEBIAN JESSIE GUEST ===="

sudo -u testuser -i lxc-create -t download -n jessie1 -- -d debian -r jessie -a amd64
#You just created an Ubuntu container (release=trusty, arch=amd64, variant=default)

#sudo -u testuser -i lxc-start -n jessie1 -d
#read press_enter_key_when_quit

if [ $? -eq 0 ]; then
  rm /tmp/log
  sudo -u testuser -i lxc-start -n jessie1 -l debug --logfile /tmp/log
  cat /tmp/log
fi

echo "==== END OF SCRIPT ===="

On pastebin, I posted the console output from all these machines:

  • Test with debian-live-7.8.0-amd64-xfce-desktop.iso
  • Test with debian-live-8.0.0-amd64-xfce-desktop.iso
  • Test with xubuntu-14.04.2-desktop-amd64.iso
  • Test with xubuntu-15.04-desktop-amd64.iso

Update

I tried further with Ubuntu 15.04 as host. I found the bug report https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1413927 Which has a similar but not same problem. But journalctl revealed problems with cgmanager:

May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/blkio/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/cpu/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/cpuset/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/devices/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/freezer/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/hugetlb/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/memory/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/net_cls/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/perf_event/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/none,name=systemd/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/blkio/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/blkio/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/cpu/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/cpu/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/cpuset/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/cpuset/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/devices/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/devices/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/freezer/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/freezer/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/hugetlb/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/hugetlb/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/memory/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/memory/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/net_cls/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/net_cls/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/perf_event/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/perf_event/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/none,name=systemd/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/none,name=systemd/user.slice/user-999.slice/session-c1.scope/lxc/jessie1

Update 2

I tried more: I installed Ubuntu trusty and vivid on two virtual machines and installed all updates on them. Then I tested both once with the original lxc package and once with the ones from the lxc daily ppa. Result is always the same error as shown above.


Eureka! Still not finished, but I managed a first time to start a container. There are a couple of things which went wrong and I found that there are many people out there having similar problems. Here is a little trouble solving guide:

Disribution / Repositories

Because of the most positive feedback, I decided to start with a Ubuntu 14.04 debootstrapped system. This means, the original system is not bigger than only a few 100M and doesn't contain many packages. I used updates and security package sources and the lxc daily ppa. Here is my /etc/apt/sources.list:

deb http://de.archive.ubuntu.com/ubuntu trusty main
deb http://de.archive.ubuntu.com/ubuntu trusty-updates main
deb http://security.ubuntu.com/ubuntu trusty-security main

deb http://ppa.launchpad.net/ubuntu-lxc/daily/ubuntu trusty main

Installation

In many tutorials, forums and bug reports I found lists of packages which have to be installed. I'm not sure yet which of those are important, but here's the list of what I finally installed (starting from a debootstrapped 14.04 system):

apt-get install bridge-utils cgmanager cloud-image-utils debootstrap distro-info \
  distro-info-data euca2ools fuse libaio1 libapparmor1 libcap2 liblxc1 \
  libpam-systemd librados2 libseccomp2 libselinux1 libselinux1 lxc python3-lxc \
  python-distro-info

Again: this list is probably longer than necessary. Here is what's really important:

  • libpam-systemd: this library is important for the correct cgroup permissions. I didn't see any error message, but without it the file /proc/self/cgroup looked like 8:blkio:/ etc. instead of 8:hugetlb:/user/1000.user/1.session
  • fuse: I read it's important for lxcfs (I would say it's a bug that lxcfs package doesn't depend on it)
  • lxc: the most important package and enough if you only use unprivileged containers
  • cgmanager: I'm still learning about its function. Btw: the lxc PPA currently provides both packages cgmanager-utils(0.27) and cgmanager(0.30). cgmanager conflicts with cgmanager-utils (<< 0.30-1) which means the two package from the same repo are not compatible (possibly a mistake). I chose cgmanager.

Configuration

I tried a lot of hints from everywhere, so I don't know yet what's important. But roughly I just ran the script from the question. Roughly it's about these files:

  • /etc/subuid
  • /etc/subgid
  • /etc/lxc/lxc-usernet
  • ~/.config/lxc/default.conf

The only additional thing I did was the following, because I found the line in many threads:

chmod +x /home/testuser/.local/share

Login as user

Now the most important thing which I still don't understand but which ruined all of my script-base experiments: Don't login using su or sudo!

I demonstrate:

root@1404-lxc-test:~# tail -1 /proc/self/cgroup
2:blkio:/user/0.user/1.session
# this is expected. I am root.

root@1404-lxc-test:~# sudo -u testuser -i tail -1 /proc/self/cgroup
2:blkio:/user/0.user/1.session
# here I used sudo to switch user

root@1404-lxc-test:~# su - testuser
testuser@1404-lxc-test:~$ tail -1 /proc/self/cgroup
2:blkio:/user/0.user/1.session
# at least here I expected the correct settings

testuser@1404-lxc-test:~$ exit
root@1404-lxc-test:~# exit

$ ssh testuser@1404-lxc-test
testuser@1404-lxc-test's password: 
Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-53-generic x86_64)

testuser@1404-lxc-test:~$ tail -1 /proc/self/cgroup
2:blkio:/user/1000.user/2.session
# now it's correct

I still don't know the reason, but it must be connected with libpam-systemd. Obviously, both su and sudo bypass PAM

And finally, don't forget to use the lxc-xxx commands when logged in as user. They will fail if you do it as root (because user-space containers are stored in .local/share/lxc/ instead of /var/lib/lxc/

Troubleshooting

These commands were most useful for me:

  • journalctl (on upstart-based releases): It revealed problems with cgmanager
  • /proc/self/cgroup
  • lxc-start using options -l debug --logfile logfilename.txt

Let me know what else is important. I think the community needs it.

Credits

Thanks to the following Tutorials:

  • https://www.stgraber.org/2013/12/20/lxc-1-0-blog-post-series/
  • https://help.ubuntu.com/lts/serverguide/lxc.html
  • https://linuxcontainers.org/lxc/getting-started/

And to these bug reports ans forum threads:

  • https://serverfault.com/questions/646176/lxc-container-not-starting
  • I'm trying to get started with LXC containers, and I'm getting a permissions problem on starting an unpriviledged container. How can I fix it?
  • https://forum.linode.com/viewtopic.php?f=23&t=11019
  • https://forum.linode.com/viewtopic.php?t=11506&p=65649
  • https://serverfault.com/questions/678984/unprivileged-lxc-container-as-root
  • plus many more

Disclaimer

I wrote this text after the first successful test. But I did so many things that some of these steps are likely not necessary. I will re-check everything on a new system soon.

Update

All the above was done with Ubuntu trusty 14.04 LTS. Now, I just tested Ubuntu wily. So far I can tell:

  • All you need to install on top of a debootstrapped base system: apt-get install lxc bridge-utils
  • No external mirrors are required. Use the lxc provided by Ubuntu
  • I didn't touch more than the following files on my fresh system: /etc/subuid, /etc/subgid, /etc/lxc/lxc-usernet, additionally, i created a user and filled its home
  • Conclusion: much easier, much more stable. For me it's the first release which really works.

(Wily is now beta and will be released on 22nd October 2015)

Related

Add a swap partition post-install 15.04
My ubuntu sony laptop connects to wifi but can't load any data
Is .profile read upon GUI login
Why does the / directory have .. and what does it mean?
Using dch to increment release number without a suffix?
watch cuts off ps aux output when piped
How to disable window effects in 15.04?
'apt-get update' gets stuck for hours at "100% [Working]" for specific PPA
How to Convert All The .MOV Videos To .MP4 in folder
By default, who is allowed to gain root privileges with pkexec?
Installing NVIDIA drivers on Ubuntu (Guest) in VirtualBox causes it to stop working
Image recognition search software?