What is the cause of this kernel panic?

I've just set up a PXE boot server on Ubuntu 14 - running kernel 3.13.0-30-generic - as it is described here https://help.ubuntu.com/community/DisklessUbuntuHowto on Supermicro X9DRFF hardware.

I have only remote access to the client server via KVM. The boot process on the client goes well but I get a kernel panic.

Is it ever possible to determine the cause of this kernel panic?

Solution 1:

Some of the output is missing, since it scrolled off the screen already, but it's possible to see that the kernel crashed in mount_root(). This means that it had a problem with mounting whatever you passed as the root filesystem. Check to ensure that you have passed the correct parameters to the kernel to boot from whatever media it is supposed to be booting from.

Solution 2:

It is actually possible. You need the debug kernel for that particular distro.

On a seperate host.

• Download the kebug version of that kernel. It will contain a vmlinux file.

Open the vmlinux file in gdb.

$ gdb /usr/lib/debug/lib/modules/3.14.9-200.fc20.x86_64/vmlinux
GNU gdb (GDB) Fedora 7.7.1-13.fc20
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/debug/lib/modules/3.14.9-200.fc20.x86_64/vmlinux...done.
(gdb) 

Judging from the stack output, we can see prior to the panic the most useful function the kernel was in was mount_block_root.

In order to determine where we failed, we need to feed in the function name plus an offset into GDB. This is done by de-referencing the address to the function, plus the offset. The stack trace supplies the offset as the first value after the function.

I.E mount_block_root+0x225 means (I was at "mount_block_root" plus 549 bytes (the hexadecimal translation).

Finally, we tell GDB to print the source code of that area. In my Linux system, this results in the following

(gdb) list *(mount_block_root+0x225)
0xffffffff81d26513 is in mount_block_root (init/do_mounts.c:422).
417                "explicit textual name for \"root=\" boot option.\n");
418 #endif
419         panic("VFS: Unable to mount root fs on %s", b);
420     }
421 
422     printk("List of all partitions:\n");
423     printk_all_partitions();
424     printk("No filesystem could mount root, tried: ");
425     for (p = fs_names; *p; p += strlen(p)+1)
426         printk(" %s", p);

From here we can tell exactly where we were at the point of the crash. NOTE my kernel is not your kernel, so the offsets are probably off. Based off of the likelihood that both these kernels are nearly the same, I'll hedge a bet that the real panic actually occurs at line 419, not line 422 (as was suggested).

Reading further up the code slightly indicates it was unable to open the block device specified -- but without a crash dump its not possible to tell why from the information. So its probably:-

• You dont want to mount a block device (likely).
• You specified a non-existent block device address (or partition).
• Your initrd, does not contain the proper filesystem module in the initrd to mount it.
• There is no filesystem on the disk.
• The superblock for the filesystem is not at the beginning of that location.

Following on from the link in you're reference, it suggests you are trying to mount with NFS as the root, in which case you should never end up landing in this function at all. In which case:

• Your kernel command line contains multiple root directives.
• You have mistyped your NFS address such that it does not get parsed correctly to go into the real function you want (mount_nfs_root).

So, overall based off of the information in the question I assume you have omitted something or made a typo.

Solution 3:

1. The quoted link does not have the complete set of parameters (append line) required for PXE booting Lubuntu 14.04.
2. The kernel panic => the mount cannot be performed correctly because of 1).

You can see How Serva solved the correct lines here (I'm related to Serva development) http://vercot.com/~serva/an/NonWindowsPXE3.html

Serva uses a CIFS share instead of NFS but you could very well use NFS if you want. Of course you do not need to use Serva; you can use its parameters in your own PXE server

[PXESERVA_MENU_ENTRY]
asset    = Lubuntu 14.04 Desktop Live
platform = amd64
kernel   = NWA_PXE/$HEAD_DIR$/casper/vmlinuz
append   = showmounts toram root=/dev/cifs initrd=NWA_PXE/$HEAD_DIR$/casper/initrd.lz,NWA_PXE/$HEAD_DIR$/casper/INITRD_N11.GZ boot=casper netboot=cifs nfsroot=//$IP_BSRV$/NWA_PXE_SHARE/$HEAD_DIR$ NFSOPTS=-ouser=serva,pass=avres,ro ip=dhcp ro

Please consider

1. Ubunu/Lubuntu have a bug that if you PXE boot them using CIFS you must add the complementary initrd INITRD_N11.GZ (freely available from Serva's page)
2. If you are installing the 64bit version the former parameters require you to rename the file \casper\vmlinuz.efi to \casper\vmlinuz