Drive reporting incorrect free space

So I swapped my shiny SATA SSD for an even shinier PCI-E SSD. I run my core OS on the SSD because it's silly-fast. I did this on my old SSD so I created a new EXT4 partition and then just dded the data across (sorry I don't know the exact command I ran anymore) and after reinstalling grub, I booted onto the PCI-E SSD. At first glance everything had worked perfectly and things were running faster than ever.

But then I noticed the free disk space on the new, larger drive: it was almost exactly the same as it was on the other disk... A disk that was half its size.

So it looks as if I've copied the files across incorrectly and it's copied some of the filesystem metadata along with it.

Tools like du and Disk Usage Analyzer come back with the correct figures. Things that look at the partition (and not the files) seem to think the drive is 120GB

I've been using this drive for a week now so it's way out of sync with the old SSD so dumping the data and starting again isn't a job that fills me with joy but two questions:

1. Is there a way to fix my filesystem so it knows what it's really on about? fsck e2fsck and badblocks all seem to be able to scan it without finding a problem with it.

2. If I do plug my old SSD back in, copy the data off my PCI-E on to it and then copy it back onto a fresh filesystem (eg juggle the data around), what's the best way of doing that? I obviously want to keep all the permissions and softlinks where they are.

The tool you used dd isn't a file copy tool, it's a disk copy tool and it copies byte for byte. This means that every piece of information, including meta data about the amount of space in the partition on the drive, has been copied.

You need to do a full backup of your files, format the drive (including a full parition format) create a new ext4 root parition and swap drive and then copy the files over.

Once you've copied the files, you need to chroot over to the new system and run the update-grub command to install the boot system on the new drive.

Or you could just run a new install of Ubuntu and put your files back.

What Oli did to fix this

1. Booted to a LiveCD and selected "Try Ubuntu"
2. Mounted my SSD and also another disk (to copy to)

3. Ran

   sudo rsync -ax /media/ssd /media/backup-drive/ssd-backup
   

   This takes a long time. I did some cleaning up before copying but still had 35GB and while I was getting write bursts of ~120MB/s, it took a while. rsync won't give you output by default but you can add --progress if you want a perverse amount of detail (although it moved too fast for me to really read it -- and probably just slowed things down).

4. I tried to nuke the drive but was getting "device busy" issues from palimpsest (Disk Utility) so I ran the installer! In the installer I just told it to create a nice filesystem using the whole SSD drive and then ran killall ubiquity after it had started copying files.

5. Then I nuked all the files that the installer had copied in, then I copied back the backup files:

   sudo rm -rf /media/ssd/*
   sudo rsync -ax /media/backup-drive/ssd-backup /media/ssd
   

6. Then I rebooted and found my UUID had changed (obvious when you think about it) so grub2 had no idea where to boot from and blew up so I headed back into the LiveCD. If you're following this as a guide, I suggest you skip this step ;)
7. Reinstall grub! I followed the help docs and followed the chroot option. You might find you need to edit your /etc/fstab while you're in there.
8. Reboot and you should be done. I'm back, I've got a metric "oodle" of free space. Huzzah.

Actually it is OK to dd the whole disk into another one. But then you need to resize the partitions to fill the new, bigger disk. You can use the awesome gparted disk partitioning tool to do this. They have their own LiveCD, too.

I assume you are doing something like:

sudo dd if=/dev/sda98 of=/dev/sda99

where /dev/sda98 has a size of 12GB and /dev/sda99 is 25GB in size.

Obviously these names are wrong, but you get the idea.

What you have done is to move not just the data but the entire file system, including all of its metadata describing what's free and what's used, to the new partition. It has lot's of free space but that free space hasn't been incorporated into the file system on /dev/sda99 so its hidden at the end of the partition and completely unusable.

The solution is to resize the file system located in the partition:

sudo resize2fs /dev/sda99

it works on EXT2, EXT3, and EXT4 file systems.

You should make a backup first.

This will tell the file system to expand into all available space on the partition, incorporating the new space into the file systems's metadata so files can be stored in it.

You can't copy a bigger partition to a smaller partition with dd unless you tell it to only copy so much with the count parameter. dd does a bit for bit copy of everything in the source partition to the target partition, in this case it tries to copy all of the hidden/invisible space to the smaller partition along with the original contents. It has no idea or care about what it is copying--it needn't be a valid file system at all.