Running containers inside a virtual machine?

It is my understanding that containers are much more lightweight than VMs because they do not virtualize the hardware but instead just isolate the software running in them from software running in other containers on the system.

My situation is such that I cannot afford to purchase additional servers, and the servers I do have are all running hypervisors. I need to provision at least 6 "servers" to host instances of relatively small web application (each instance is for a different customer). It would be wasteful to provision a new VM for each of these servers, so I plan to set up 2 VMs and do one of the following:

1. Run multiple instances of the application on each VM using virtual hosts (apache). or
2. Set up containers on the VMs which would allow me to isolate the environments.

Considering this, is there a non-negligible overhead associated with container systems that would make it a bad idea to use them inside a VM, or is there any other technical reason why I would not want to use containers?

Solution 1:

Docker is VERY lightweight compared to a VM and a VM system should function just fine running containers. Each container essentially does run as an isolated system so it's very good for isolation from a perspective of system stability. Based on your description it sounds like the ideal use case for Docker. If you do experiment with Docker make sure you use the newest version possible, some of the older ones have some fairly nasty vulnerabilities in them. There are some security considerations when running Docker.

SELinux - SELinux is container aware, and will automatically create a randomly named MCS label for each container. This helps ensure isolation as LXC containers are not considered to truly "contain" by themselves, though this is improving.

USER Directive - In each Dockerfile it's recommended best practice to use the USER directive and have the user run as an account other than root, the default. The catch here is the user must exist on the system. It also can be frustrating chowning/chmodding files and directories for this new user, but it helps reduce your risk. I typically recommend that you create a "containeruser" or something like that on your systems to help ensure that a common user is available on all systems that doesn't overlap with any other part of the system.

Otherwise the toughest part is managing the containers and updating them when necessary.

Solution 2:

is there a non-negligible overhead associated with container systems that would make it a bad idea to use them inside a VM

Does Docker Containers Performance in VMware vSphere help you?

or is there any other technical reason why I would not want to use containers?

I don't know about Docker in general because I haven't worked with it yet. I think that the management software still lacks maturity compared to VM management software... but I'm a vSphere administrator and therefore probably biased.

Considering Docker on VMs, CoreOS is now officially supported on vSphere 5.5. So at least VMware thinks that Docker / containers on VMs is OK.