Kernel upgrade CentOS 5.3 mount: could not find filesystem '/dev/root'

We have a CentOS 5.3 x64 server that by default runs kernel version 2.6.18-164.11.1 and we are attempting to upgrade the box to 2.6.31.12 The drive is LVM +ext3, and the problem I'm having is when I upgrade the kernel and attempt to boot from it, no matter what version of the kernel I use, I get /dev/root not found towards the end of the boot process, and the kernel panics, and than reboots.

I'm installing the kernel exactly as it says in this doc. I've tried it "The centOS way " using make rpm and than installing that. I've updated my mkinitrd. The most interesting part of this problem is that it has been so frustrating that I decided to try and clean install centos on an identical machine without LVM, and the result is EXACTLY the same. After upgrading the kernel, I get /dev/root not found. Does anyone know how to fix this, or what information would be relevant to remedy it? I'm open to try anything at this point.

One more interesting thing about this problem is that in the new version of the kernel, during boot it complains that dm-mapper is started twice, than panics right after that. I've tried this with other kernel versions, and the result is the same. What am I missing here?

If you need any more files, please just ask.

 Linux cg 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:32:21 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

  /dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
    LABEL=/boot             /boot                   ext3    defaults        1 2
    tmpfs                   /dev/shm                tmpfs   defaults        0 0
    devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
    sysfs                   /sys                    sysfs   defaults        0 0
    proc                    /proc                   proc    defaults        0 0
    /dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.31.12-rt20) //NOT WORKING!!!!
        root (hd0,0)
        kernel /vmlinuz-2.6.31.12-rt20 ro root=/dev/VolGroup00/LogVol00 isolcpus=8,9,10,11,12,13,14,15 panic=10
        initrd /initrd-2.6.31.12-rt20.img
title CentOS (2.6.18-164.11.1.el5) //WORKING!!
        root (hd0,0)
        kernel /vmlinuz-2.6.18-164.11.1.el5 ro root=/dev/VolGroup00/LogVol00 isolcpus=8,9,10,11,12,13,14,15 panic=10
        initrd /initrd-2.6.18-164.11.1.el5.img

Solution 1:

I am going to add my previous comment as an answer here as this has gone to bounty. Apologies for the blatant attempt to score reputation points.

I had this exact same problem and resolved it by changing the setting of

CONFIG_SYSFS_DEPRECATED_V2=Y

in the .config file. as mentioned in this question by godpapa.

Just search for CONFIG_SYSFS_DEPRECATED_V2 and change it to Y and rebuild the RPMs. Then install the RPM.

I can confirm this does fix the problem on CentOS 5.4 when building a 2.6.33 kernel using the "The CentOS way". I would be surprised if it didn't work for 2.6.31.12 on CentOS 5.3 too.

Solution 2:

Try to enable CONFIG_SYSFS_DEPRECATED_V2=y