What's the proper way to prepare chroot to recover a broken Linux installation?

This question relates to questions that are asked often. The procedure is frequently mentioned or linked to offsite, but is not often clearly and correctly stated. In an objective to concentrate useful information in one place, this question seeks to provide a clear, correct reference for this procedure.

What are the proper steps to prepare a chroot environment for a recovery procedure?

In many situations, repairing a broken Linux installation is best done from within the installation. But if the system won't boot, how do you fix it from within?

Let's assume you manage to boot into an alternate system. Once there, you need to access your broken installation in order to fix it. Many recovery How-Tos recommend using chroot in order to run programs as if you are actually booted into the broken installation.

What is the basic procedure?
Are there accepted best-practices to follow?
What variables need to be considered in order to adapt the basic preparation steps to a particular recovery task?

As this is Community Wiki, feel free to edit this question to improve it as well.

Solution 1:

Here are some resources:

ArchLinux wiki on "Change Root"
ArchLinux Wiki on "Reinstalling GRUB"
Gentoo Wiki on "Chroot from a livecd"

"Changing root" or "chrooting" is a method for zooming in on part of your filesystem, so that, for example, /path will refer to what was formerly accessible at /mnt/path. The "root" in the expression "chroot" refers to the root filesystem /, not to the root user. (Though typically you will need root user privileges in order to chroot.)

Preparation

All the steps in this guide will have to be performed as root user.
We suppose your hard disk is at /dev/sda1 and its filetype is ext3. If you don't know the location and filetype of your disk, read the output of fdisk -l.
Be sure that the architectures of the system you booted into (e.g. it's a 32bit LiveCD) and the system you wish to enter (e.g. it's a 32bit installation on your hard drive) match. You can determine the architecture you booted with using uname -m.
Be sure any kernel modules you need have been loaded.
Set up your network if you need it (e.g., to install updated packages).
Initialize your swap partition if necessary (e.g., swapon /dev/sda3).

Performing the chroot

cd /
mount -t ext3 /dev/sda1 /mnt
mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -o bind /dev /mnt/dev

If your /boot directory is on a different partition from your /, and you want to manipulate files on it (e.g., if you'll be working with GRUB, performing a kernel upgrade, etc.), you'll also need to mount that partition. If it's at /dev/sda2 and its filetype is ext2, then do:

mount -t ext2 /dev/sda2 /mnt/boot

Similarly for any other parts of your filesystem (/var, /usr) that reside on separate partitions but which you need access to. Generally when you're chrooting to fix something you won't need access to /home, so you don't need to bother with it.

(It's also possible to mount filesystems after you've chrooted, but it's smarter to do so beforehand. The reason is that when you do it after, the outside/kernel environment won't know about the mounted filesystems, so if you forget to umount them before exiting the chroot, the system won't know to umount them when it shuts down, either. This could damage those filesystems.)

If you've setup your network and want to use it in the chrooted system, copy over /etc/resolv.conf so that you'll be able to resolve domain names:

cp -L /etc/resolv.conf /mnt/etc/resolv.conf

Now you're ready to move into the mounted filesystem:

chroot /mnt /bin/bash

(If this returns an error chroot: cannot run command '/bin/bash': Exec format error, this usually indicates that you booted with one architecture (e.g. x86_32) and are trying to chroot into another (e.g. x86_64). The solution is to use a LiveCD which has the same architecture as the system you want to chroot into.)

At this point, you're still running the kernel you booted with, but all paths /path will refer to what used to be /mnt/path.

If you'll be doing anything with GRUB, you'll need to be sure your /etc/mtab file is up-to-date:

grep -v rootfs /proc/mounts > /etc/mtab

It might also be helpful at this point to do:

source /etc/profile
export PS1="(chroot) $PS1"  # add a reminder to your prompt

Do your dirty work

At this point, you can perform whatever troubleshooting you need to do:

resintall GRUB to your disk's MBR
reset a forgotten password
perform a kernel upgrade (or downgrade)
rebuild your initramdisk
fix your /etc/fstab
reinstall packages using your package manager
whatever

Cleaning up

When you're finished, ensure that all running programs have stopped. Then exit the chroot:

exit

Now unmount all the partitions you mounted:

umount /mnt/boot # if you mounted this or any other separate partitions
umount /mnt/{proc,sys,dev}

Finally attempt to unmount your hard drive:

umount /mnt

If you get an error saying that /mnt (or any other partition) is busy, this can mean one of two things:

A program was left running inside of the chroot.
Or more frequently: a mount point still exists on this mount. For example, /mnt/usr is still mounted when trying to unmount /mnt.

In the latter case, simply unmount the offending mount point first. To get a reminder of all the current mount points, run mount with no parameters.

Finally:

reboot