Linux containers (LXC) on Red Hat/CentOS EL6 - lxc-create versus libvirt?

It's tricky trying to stay within the good graces of Red Hat and still plan for system longevity...

I've been a proponent of Linux Containers (LXC) for over a year. My initial installations were based on information gleaned from online tutorials, like this one and this one. This centered around the lxc-create, lxc-start|stop and lxc-destroy commands and modifying existing OpenVZ templates.

This works well and is happily running in production. However, I am bringing up some additional systems and decided to check Red Hat's current documentation regarding containers in EL6. I was surprised to see their official stance on this.

In Does RHEL 6 provide LXC tools needed to use Linux Containers?, Red Hat describes LXC as a Technology Preview and suggests using libvirt to manage create and manage containers.

Yet Oracle advocates a totally different containerization technique in its Unbreakable Linux.

There appears to be some missing functionality in the libvirt method, but my initial approach with lxc-* commands was a bit of a manual process... I can't quite tell what's right or the "accepted" means of managing containers on EL6.

• What's the conventional wisdom regarding LXC and RHEL-like systems today?
• How are you implementing them in your organization?
• Are there any advantages to one approach versus the other(s)?
• Can these coexist?

What's the conventional wisdom regarding LXC and RHEL-like systems today?

Personally, I find the current setup somewhat lacking. LXC seems more at the forefront -- certainly more maintained.

How are you implementing them?

In terms of offering it as a virtualization option I am not. I find the current technological setup lacking.

• No username namespace.
• Certain mountpoints are not namespace aware (cgroups, selinux)
• Values in /proc are misleading system globals that dont account for the resource partitioning in namespaces.
• Breaks audit.

I do find it really nice tool for application level containment however. We use namespaces and cgroups directly to contain network and IPC resources for certain user-ran web applications. We provide our own interface to control it. In RHEL7 I'm considering moving this functionality to libvirt-lxc as the newer revisions of libvirt support the concept of user ACLs.

For virtualization in terms of a fully initialized system I'm waiting to wee what is offered in RHEL7, but in all honesty, I feel we might only see a good enough solution once we're on a later minor release of RHEL7 and then perhaps only on a technology preview state.

Keep your eye out on systemd-nspawn something tells me in the next 18 months or so it might take its place is the best tool to do fully linux contained virtualization, be it that the systemd authors make it clear its not secure right now! I wouldnt be surprised if libvirt drops libvirt-lxc eventually and just offers a wrapper around systemd-nspawn with systemd slices defined.

Also, be wary theres been a lot of talk over the last 6 months in regards to re-implementing cgroups as a kernel programmer interface rather than a filesystem interface (perhaps using netlink or something, haven't checked) so systemd should be very hot on the tail of getting that right very quickly.

Are there any advantages to one approach versus the other?

I think the LXC option (not libvirt-lxc) is better maintained. Having read the libvirt-lxc sourcecode, it feels rushed. Traditional LXC certainly has newer features whicih have been better tested. Both require a degree of compatibility by the init system being ran in them, but I suspect that you'll find LXC slightly more "turn-key" than the libvirt-lxc option particularly in regards getting distros to work in them.

Can these coexist?

Sure, remember that for all intents and purposes, both are doing the same thing. Organizing namespaces, cgroups and mount points. All the primitives are dealt with by the kernel itself. Both lxc implementations just offer a mechanism for interfacing with the kernel options available.

Red Hat is making a huge containerization push. They're building an entire new product, Red Hat Enterprise Linux Atomic Host, around it.

For a less radical approach, take a look at their RHEL7 beta Resource Management and Linux Containers Guide; you'll notice it pushes libvirt-lxc and makes no mention of the lxc tools.

The lxc-* executables are packaged in the lxc package in EPEL. However it's the old "long term support" release. You don't even have the "-f" option in lxc-ls. I'd simply install Ubuntu for my LXC hosts.

The RHEL way to manage LXC seems to be via libvirt-lxc but it's apparently deprecated.

Noted that Ubuntu is supporting many of the new lxc/lxd development while Redhat is focusing in KVM and docker.