Understanding Why I have lost so much space mounting a new HDD
I've just purchased a 4TB Western Digital Red hard disk for my Media Server. I want to understand why I've ended up losing so much space by formatting it as ext4 and then mounting it. I understand some will be lost for the drives own files but what I have lost seems a bit much.
Steps I have taken:
Formatted the Drive using sudo mkfs.ext4 /dev/sde
Edited FS tab so it auto mounts at boot
UUID=3276bc13-20a5-4225-ad33-d8fecdd5216c /wd_red_1 ext4 defaults 0 0
Now when I run df -h this is my output
/dev/sde 3.6T 89M 3.4T 1% /wd_red_1
Does it not seem a bit much that in total I have lost 700GB of space?
In comparison I've got a seagate 2TB HDD which I've been through the same process with and it's only taken 200GB (so I have 1.8TB usable). Have I done something wrong?
Server is running 18.04.3 LTS
Solution 1:
Disc manufacturers list the size in decimal values, so your 4TB HD is actually 4,000,000,000,000 Bytes. But df -h lists in binary base numbers (i.e. 4KB is 4096 Bytes, 4MB is 4,194,304 Bytes, 4GB is 4,294,967,296 Bytes, and 4TB is 4,398,046,511,104 Bytes. You only bought a disc with 4,000,000,000,000 Bytes, so that works out to 3.63798807 TB (in the binary number base). This is pretty much exactly the value that you're seeing listed by df -h.
You didn't lose any capacity, it's just a numbers game played by the manufacturers.
To check this, try df -H. With a capital H, it lists it in decimal values, so instead of "3.6T" it will say "4T" or very close to that number. The manufacturers even go further and the "4TB" they list is also approximate. The actual may be slightly more or less than that.
Solution 2:
For sure the accepted answer provided by Rich is the main reason for the observed difference between the specs of the manufacturer and what df shows. However, there are some factors that cause you to really have less space than what would appear from the capacity of the disk.
Part is due to the overhead of ext4 file systems. These systems are very reliable and robust. The structures of the file system need considerable space. You can read about the technical details here (with thanks to user Matt).
Another reason is the "reserved space" taken, which is a fixed percentage of the total drive capacity. Typically, that fixed percentage is way to high for large drives and especially for data drives. Linux systems reserve typically 5% of new partitions for the root user and system services. The idea here is even when you run out of disk space, the root user should still be able to log in and system services should still run. This cannot happen if there is no space left on the root partition.
You therefore will gain space by decreasing this reserved space. With tune2fs, you can adjust that percentage:
sudo tune2fs -m 1 /dev/sdb1
would reduce the reserved space to 1% on a partition /dev/sdb1 (substitute your own partition here). For a data drive, you probably could set this to zero. However, to prevent too much data fragmentation to build up, an ext4 volume should never be fully filled, so besides to keeping under the maximum storage, you can also leave some reserved space, which also contributes to the remaining free space needed to avoid fragmentation.