Why doesn't my sparsebundle report free space, with plenty to spare?
I have a Time Machine created sparsebundle (HFS+) that reports the following limits when using hdiutil resize -limits
:
min cur max
403186088 403186088 34359738368
That's a current size of 206 GB and a max of 17.5 TB. I draw the conclusion that I don't need to hdiutil resize
the sparsebundle to allow it to expand. The host volume has an additional 200 GB free, so the bundle should therefore be able to expand about 100% -- however:
- Disk utility reports free space as 1 GB
- Time Machine reports free space as 1 GB
I expected them to detect 200 GB of free space available on the host volume. This is a problem since Time Machine will start deleting backups prematurely.
How do I make the sparsebundle expand when necessary to fill available space on the host file system?
Some additional questions that came up that could help to narrow it down:
- Which part of the system is responsible for expanding the bundle when required? Does Time Machine do this explicitly or is it done transparently underneath it? I.e. is this a problem with Time Machine or with something more fundamental?
- Is there a more reliable way to check whether Time Machine has/will run out of space than to look at the "free space" figure in Time Machine settings? I.e. can the numbers be trusted? I did remove and add the bundle as a backup destination with
tmutil setdestination
before checking free space. - If Time Machine is explicitly responsible for expanding a sparsebundle used as a backup target, will it only do this when it itself mounts the sparsebundle as a step of a networked backup? In other words, when taking the networked drive and plugging it in locally with USB3, and then manually mounting the sparsebundle, would Time Machine just skip the step where it usually expands the sparsebundle target since the bundle is now considered local storage (which doesn't generally use sparsebundles)?
MacOS Mojave 10.14.6
Update
There are several subtle reasons that combine into why I think my sparsebundle doesn't expand automatically, and why I form the null hypothesis that the free space reported by Time Machine is not erroneous. They include:
- I have no good reason to assume that the reported number in Time Machine and Disk Utility UI is erroneous. Conversely, I have been given no explicit assurances that the target volume will expand, nor to what extent.
- Time Machine has to my recollection failed previously to expand the sparsebundle target that it was backing up to, despite lots of available space on the host volume.
- Apple's track record of design suggests to me that if they only expose a single figure (remaining space available) in the Time Machine UI for me to base my decisions on with regards to what actions TM will take in turn, that it would've been removed already if it were an unreliable figure. I don't count the use of sparsebundles (i.e. networked backup) as such a small corner case as to warrant that level of unreliability to be ignored in their UI design phase.
- When the drive is used as a network target, TM seems to report ~200/400 GB available, while attached as a local backup target it will report ~200/200 GB available. A locally mounted sparsebundle is also not showing up in the "add target" UI in TM, but has to be added using
tmutil
. This suggest an inconsistency in how backup targets are handled and that Time Machine manages sparsebundles explicitly.
I realize this is not conclusive, so I tried it out:
- I mounted the host volume over USB, and proceeded to resize the sparsebundle with
hdiutil
to be as small as I could make it, so that it appeared to have ~20 GB free according to all system GUI tools. It shrank by over 100 GB in that regard, so plenty of space to spare on the host volume. - I mounted the bundle as a local volume, and proceeded to copy an 11 GB Xcode.xip file to it twice.
- The first transfer succeeded.
- The second transfer failed before even starting -- "not enough free space". 1.54 GB of additional space was required. This was expected.
- Conclusion: the sparsebundle did not expand transparently, despite over 100 GB of free space on the host volume. There is no reason for me to believe that the figure that Time Machine reports is erroneous, nor that the volume will expand transparently during TM operation.
Regarding your first question: It is the operating system that automatically resizes sparse bundles as necessary when you add data to them. This is done transparently underneath Time Machine, so it is not something that Time Machine explicitly handles.
The sparse bundle itself can have a maximum capacity that exceeds the available space on your drive, as it does not actually occupy any space before you write data to it. You can change resize this, but it doesn't actually change anything about how much space it takes up on the physical drive, but rather just imposes a limit on how much can eventually be written to it.
Even though Disk Utility reports it as having no real free space, it will automatically expand to fill up the drive. When the physical drive is filled up, it is ofcourse not possible to add more to the sparse bundle until you clear up space or move the bundle to a larger drive.
When you run hdiutil resize -limits my.sparsebundle
you'll get a min, cur and max value. You can use the following calculations to get "human-readable" numbers for them:
min * 512 / 1024 / 1024 = MB data stored in the sparsebundle
cur * 512 / 1024 / 1024 = limit in MB of how much data can be added
max * 512 / 1024 / 1024 = the largest limit in MB you can set
You can change the limit at any time using:
hdiutil resize -size 500MB my.sparsebundle
This would change the limit to 500 MB. Obviously it doesn't make sense to set the limit lower than the current amount of data stored. Perhaps a bit counter intuitive, the limit can be set higher than the amount of disk space you actually have available. The limit cannot however be set higher than the max
value, which in your case is 16 TB. The reason for the 16 TB is that you have it setup with a block size of 4 kB. You can create sparse bundles with a larger block size to get a higher max value up to the theoretical maximum of just shy of 8 exabytes.
Note that this limit does not have anything to do with how much disk space is actually taken up on your physical disk. You can find this number running for example:
du -sh my.sparsebundle
If for example use Disk Utility to create an empty sparsebundle with a size of 500 MB formatted with APFS, you'll see that the above command tells you that it takes up only approx. 13 MB on the physical disk.
When you mount the sparsebundle and add files to it, the amount of space taken up on the physical disk increases. I.e. the operating system transparently expands the sparsebundle structure on the physical disk to accommodate the extra data you add to it. However, it won't add more than the arbitrary limit, you have set it.
If you select "Info" on the mounted sparsebundle in Finder, it will show you how much data has been added to the sparsebundle as well as the amount of "Free" space. This amount of "Free" space is not the amount of free space on your physical drive, but rather how much is left until you hit the (artificial) limit set on your sparsebundle, which you can change at any time later.