How can I make a SquashFS file the root file system when doing an iPXE boot?

I have a system that works perfectly fine as iPXE boot server. Now I'm trying to make it use a SquashFS file as a root file system, instead of an NFS file system.

The current system uses iPXE configuration as the following (relevant lines shown), and it's working fine:

:retry
kernel http://${next-server}/installdcos/ubuntu_os/vmlinuz root=/dev/nfs nfsroot=${next-server}:/installdcos/ubuntu_os/nfsroot ro ip=dhcp BOOTIF=%(mac)s console=ttyS0,38400n8 console=ttyS1,9600n8 console=tty0  || goto retry
initrd http://${next-server}/installdcos/ubuntu_os/initrd.gz || goto retry
boot || goto retry

Now, instead of the booted system use /installdcos/ubuntu_os/nfsroot as the root file system, I want it to use a single SquashFS as root file system (I mean the system should mount it and then use it as the root file system).

For this, first I've created a file ubuntu_os.squashfs that holds the contents of nfsroot directory using mksquashfs. Then I've placed this ubuntu_os.squashfs file in /installdcos/ubuntu_os/nfsroot directory.

Now, in my iPXE configuration, I want to say: please get the ubuntu_os.squashfs via network, mount it as a SquashFS file system, and use it as your root file system (so that you will not have go over NFS again for invoking commands that reside in the root file system).

But when I try the following change for the kernel line in the configuration:

kernel http://${next-server}/installdcos/ubuntu_os/vmlinuz root=/dev/nfs nfsroot=${next-server}:/installdcos/ubuntu_os/nfsroot/ubuntu_os.squashfs ro rootfstype=squashfs ip=dhcp BOOTIF=%(mac)s console=ttyS0,38400n8 console=ttyS1,9600n8 console=tty0  || goto retry

And do an iPXE boot from a machine, it starts the iPXE boot process, and after printing some lines, it gets stuck at the following error:

  Begin: Retrying nfs mount ... Begin: Running /scripts/nfs-premount ... done.
  mount: Not a directory
  done
  Begin: Retrying nfs mount ... Begin: Running /scripts/nfs-premount ... done.
  mount: Not a directory
  done
  Begin: Retrying nfs mount ... Begin: Running /scripts/nfs-premount ... done.
  mount: Not a directory
  done

The error message makes sense, indeeed, /installdcos/ubuntu_os/nfsroot/ubuntu_os.squashfs is not a directory.

So what kind of parameter and value combination should I write for that kernel configuration line?

I've checked documentation such as http://www.tldp.org/HOWTO/BootPrompt-HOWTO-3.html and https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt, but could not figure out what magic incantation I should pass to kernel (and the rest of the SquashFS boot tutorials I found consider the boot operation taking place from disk, and not PXE, or doing it LiveCD, which is not exactly like my case).

For understanding what's going on you need to understand how Linux PXE boots.

1. The kernel vmlinuz is transferred
2. The initrd initrd.gz is transferred
3. The kernel mounts initrd and starts its init script

initrd is an ultra-minimal Linux system containing the minimal functionality for connecting to (NFS) or retrieving (HTTP/CIFS) the "real" OS (in your case contained within ubuntu_os.squashfs), mounting it and finally "chrooting" into it.

A PXE ready init script is in charge of parsing the kernel variables, starting the net services, handling NFS, HTTP, CIFS, etc. In most of the cases a particular initrd.gz is not able to deal with all the protocols mentioned above then you need to "customize" your initrd.gz or to create a "complementary" initrd providing the missing features when needed.

In your case your init script still think it has to mount an NFS directory and not a file; then if you want to use NFS you should mount the directory where your ubuntu_os.squashfs is located and next mounting the ubuntu_os.squashfs file. THis of course means patching init (and/or its associated components)

If you do not mind offering ubuntu_os.squashfs on a CIFS share you can see what Serva does for PXE booting Ubuntu live distros; you will see there all the parameters for CIFS booting (I'm related to Serva development)

i.e. Ubuntu LTS 14.04 Desktop Live

[PXESERVA_MENU_ENTRY]
asset    = Ubuntu LTS 14.04 Desktop Live
platform = amd64
kernel   = /NWA_PXE/$HEAD_DIR$/casper/vmlinuz
append   = showmounts toram root=/dev/cifs initrd=/NWA_PXE/$HEAD_DIR$/casper/initrd.lz,/NWA_PXE/$HEAD_DIR$/casper/INITRD_N11.GZ boot=casper netboot=cifs nfsroot=//$IP_BSRV$/NWA_PXE_SHARE/$HEAD_DIR$ NFSOPTS=-ouser=serva,pass=avres,ro ip=dhcp ro