How to reclaim all/most free space from a sparsebundle on OS X

Is there a possibility to "defragmentize" a sparse bundle image and reclaim (most) of the free space?

Here is the background: I am using sparse bundles and every now and then I want to reclaim space from them so I run:

hdiutil compact image.sparsebundle

However, as explained in the man page, it only reclaims completely unused band files, so in my case it says:

Reclaimed 0 bytes out of 90.4 GB possible.

Of course there is the possibility to copy the contents of this image to a new sparse bundle that is then used in lieu, but that is both cumbersome and requires enough free space for this operation.

Meanwhile, I found out that the output of the compact command is somewhat misleading (I am currently running OS X 10.5.7) as it sometimes lists a size as possible that is larger than the size currently taken up by the image bundle on the hard drive. I did not look closer but the output seems to be either the maximum size or "maximum size" - "used size".

Interesting!

From what I've heard, the sparse bundle divides the data into 8Mb bands. Changing the band size might just help, if you're lucky. I mean, you'll never get 100% reclaimed space, but maybe better than what you get now. (Depending on the data on the image etc.)

I did a dirty simple test with two 500Mb sparse bundles, one with 8Mb (default) band size, and one with 1Mb (smallest allowed size from what I can tell). I copied over 400mb of mp3 files and then removed every other file and then run hdiutil compact on their asses.

Size after compact
8Mb bands: 271Mb
1Mb bands: 215Mb

The command to convert your sparse bundle is

hdiutil convert src.sparsebundle -format UDSB -tgtimagekey sparse-band-size=2048 -o dst.sparsebundle

Band size is in the unit 512byte. So the above example sets the band size to 512 * 2048 = 1Mb. Just be careful if you're dealing with TimeMachine images or user home folder images etc. You're deviating from the Apple path :) Keep a fail safe backup!

As for defragmentation: I have a funny feeling it's just as fast (or faster!) to just use hdiutil to convert the sparsefile to a new sparse file with the same format. I think it tries to be smart about it. But I don't know.

(Note that defragmenting a sparse bundle just defragments the disk data, not the sparse bundle bands, unless it's a sparse bundle aware defragmentor. hdiutil convert does a band 'defragmentation' I believe.)

I think the 90.4 GB possible is plain wrong. When I ran:

hdiutil compact *.sparsebundle

...it freed up 16 GB of space, which is roughly what I expected. Funnily enough it said roughly "800 GB possible" which is way more than the capacity of my HD. So I assume the second figure is just some (flawed) theoretical number.

@Oscar pointed us in the right direction. But you don't need to convert to 1 MB and compact, hoping to regain space.

Instead, just copy your sparsebundle to another new sparsebundle. This, in turn, defrags the image for you. And at the same time you can change filesystem, size and even the bands.

Here's an example I did recently, converting my 20GB "code" image that i sync across devices.

$ hdiutil create -size 20g -type SPARSEBUNDLE \
-imagekey sparse-band-size=2048 -fs HFSX \
-volname code -attach ~/sync/images/code.sparsebundle

See all options with hdiutil create -help.

This command outputs:

/dev/disk3              GUID_partition_scheme
/dev/disk3s1            EFI
/dev/disk3s2            Apple_HFS                       /Volumes/code
created: /Users/eric/sync/images/code.sparsebundle

You can see the -attach option mounted it under /Volumes/code.

Note: I use HSFX for case-sensitive HFS+ because I sync my images to Linux as well and mount them. You'll most likely want to use HFS+ instead, since macOS is normally case-insensitive.

Now all I have to do is copy everything to my new sparsebundle:

$ cp -r /Volumes/IMG_CODE/ /Volumes/code/

And now to compare:

# old sparsebundle
$ du -h ~/sync/images/IMG_CODE.sparsebundle/
15.0G   /Users/eric/sync/images/IMG_CODE.sparsebundle/bands
15.0G   /Users/eric/sync/images/IMG_CODE.sparsebundle/

# new sparsebundle
$ du -h ~/sync/images/code.sparsebundle/
3.0G    /Users/eric/sync/images/code.sparsebundle/bands
3.0G    /Users/eric/sync/images/code.sparsebundle/

Yay. My 15GB sparsebundle is now 3GB, defragged, and bands changed!