Amazon EC2 instance won't boot: Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Solution 1:

I created a new instance by going to Images > AMIs > Private Images > Selecting the image the instance was started from > Launch. I launched in exactly the same availability zone, not just US or region, but the 2a, 2b, 2c must match as well. I stopped the new instance. I disconnected the EBS volume from the old instance. I re-attached the EBS volume to the new instance at /dev/sdf. I started the new instance. The EBS volume shows up inside the instance as /dev/xvdj so I mounted it with mkdir /xvdj; mount /dev/xvdj /xvdj. I edited /xvdj/boot/grub/grub.conf and changed default=0 to default=1. I saved the file, stopped the new instance, re-attached the EBS volume to the old instance and it started. I ran yum update in the old instance and double-checked /boot/grub/grub.conf and double-checked that it would reboot.

I also found this regarding updates to CentOS kernel: grub.conf missing initrd path after kernel update

I noticed after I ran yum update I now had 2 entries in grub.conf without initrd. Running # yum reinstall kernel.x86_64 works to fix that.

Solution 2:

I've had this same issue on several occasions and had to solve it by restoring the instance from EBS snapshot backups. Today I had the same issue and was determined to resolve it without having to restore from backups. I did the following:

1. Detach the root volume from the failed instance /dev/sda1.
2. Attach the volume into a working instance and mount the volume (e.g. mount /dev/xvdh /xvdhmount)
3. Back up the boot folder: mv /xvdhmount/boot /xvdhmount/boot-backup
4. From a working instance with the same version of OS in my case RHEL 7.4 copy the entire contents of the /boot folder via SCP or WinScp into /xvdhmount/.
5. Detach the volume from the working instance and attach back to the failed instance.
6. Start the failed instance .... the instance did boot and I am able to log in.

I hope this helps!

Solution 3:

I had a similar problem with a CentOS instance. This AWS support article gives quite a good overview. Here's how I and managed to solve my problem:

• Shut down the original EC2 instance and then detach the /dev/sda1 disk
• Start a new, temporary EC2 instance, and attach the disk as /dev/sdp to the new EC2 instance
• SSH into the new EC2 instance and mount /dev/sdp to /data

Then I wanted to go back to a previous kernel. The instructions on the CentOS wiki were helpful:

• List all Grub entries with grep "^menuentry" /data/boot/grub2/grub.cfg | cut -d "'" -f2
• Picked the 2nd one from the top, in my case this was CentOS Linux (3.10.0-957.12.1.el7.x86_64) 7 (Core)
• Configure the boot default with grub2-set-default --boot-directory /data/boot/ 'CentOS Linux (3.10.0-957.12.1.el7.x86_64) 7 (Core)'

Then shut down the new EC2 instance, detach the volume, attach it back to the original instance (to /dev/sda1) and boot the initial instance.

Solution 4:

Me Too!

The underlying cause was an interrupted yum upgrade and a junior staffer doing the work reconnected, and ran yum-complete-transactions to finish everything.

However, something didn't write a file into /boot/initrd....newver.... which was probably related to the latest kernel entry in grub2.cfg missing its initrd=/.... line completely.

The quick fix was to reattach the boot disk volume to a different instance, mount it, and edit /mountpoint/etc/grub2.cfg so that the instance starts up the older version of the kernel. Then re-disconnect and reattach to /dev/sda1 of the original instance.

NOTE lately its been hard to attach a centos boot volume to a different centos machine, because the UUID is the same on the root volume. Workaround is to use a different OS as your temp machine, like Debian for CentOS disk fixups.

Once you're in again, run yum reinstall kernel* to repeat the missing steps, and on completion reboot again to be sure it restarts properly this time and onto the newest kernel.