Missing space on ZFS (Nexenta)

I am facing an issue where ZFS appears to be missing around 4TB of space on a 16TB volume.

The ZFS pool is reporting the correct size (16.3TB):

nmc@thn-nstor1:/$ zpool list
NAME           SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
nstore1pool1  16.3T  15.2T  1.11T         -    93%  1.00x  ONLINE  -
syspool         29G  9.20G  19.8G         -    31%  1.00x  ONLINE  -

The zfs list command however reports 4TB less:

nmc@thn-nstor1:/$ zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
nstore1pool1               12.4T   770G   116K  /volumes/nstore1pool1
nstore1pool1/.nza-reserve  49.8K   770G  49.8K  none
nstore1pool1/nstore1zvol1  28.4K   770G  28.4K  -
nstore1pool1/veeam         12.4T   770G  12.4T  /volumes/nstore1pool1/veeam

Further to reading this post I also ran zfs list -o space and zfs list -t snapshot to verify that there are no snapshots using space, which confirm the results above:

nmc@thn-nstor1:/$ zfs list -t snapshot
NAME                             USED  AVAIL  REFER  MOUNTPOINT

nmc@thn-nstor1:/$ zfs list -o space
NAME                       AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
nstore1pool1                770G  12.4T         0    116K              0      12.4T
nstore1pool1/.nza-reserve   770G  49.8K         0   49.8K              0          0
nstore1pool1/nstore1zvol1   770G  28.4K         0   28.4K              0          0
nstore1pool1/veeam          770G  12.4T         0   12.4T              0          0

** EDIT **

Further to requests for more info, here is the output of zfs status -v and zfs list -t all (abridged for brevity):

nmc@thn-nstor1:/$ zpool status -v
  pool: nstore1pool1
 state: ONLINE
  scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        nstore1pool1               ONLINE       0     0     0
          raidz1-0                 ONLINE       0     0     0
            c0t5000C5003F0DE915d0  ONLINE       0     0     0
            c0t5000C5003F17FA16d0  ONLINE       0     0     0
            c0t5000C500506272F7d0  ONLINE       0     0     0
            c0t5000C50063E7E297d0  ONLINE       0     0     0
            c0t5000C500644D6CE0d0  ONLINE       0     0     0
            c0t5000C500644FBA98d0  ONLINE       0     0     0
            c0t5000C500644FFD61d0  ONLINE       0     0     0
            c0t5000C50064509003d0  ONLINE       0     0     0
            c0t5000C50064AE3241d0  ONLINE       0     0     0
          raidz1-1                 ONLINE       0     0     0
            c0t5000C50064BF602Dd0  ONLINE       0     0     0
            c0t50014EE65BA8D06Dd0  ONLINE       0     0     0
            c0t50014EE6B0FE0EA6d0  ONLINE       0     0     0
          raidz1-2                 ONLINE       0     0     0
            c0t50014EE606A6F0E5d0  ONLINE       0     0     0
            c0t50014EE65BFCD389d0  ONLINE       0     0     0
            c0t50014EE65BFD0761d0  ONLINE       0     0     0
            c0t50014EE65BFD11A3d0  ONLINE       0     0     0
            c0t50014EE6B150B5FBd0  ONLINE       0     0     0
            c0t50014EE6B152CB82d0  ONLINE       0     0     0

errors: No known data errors

nmc@thn-nstor1:/$ zfs list -t all
NAME                             USED  AVAIL  REFER  MOUNTPOINT
nstore1pool1                    12.4T   770G   116K  /volumes/nstore1pool1
nstore1pool1/.nza-reserve       49.8K   770G  49.8K  none
nstore1pool1/nstore1zvol1       28.4K   770G  28.4K  -
nstore1pool1/veeam              12.4T   770G  12.4T  /volumes/nstore1pool1/veeam

I would appreciate any help in understanding where this missing space has gone?


Solution 1:

Oh my god... what did you do?!?

In general, RAIDZ1/2/3 will show zpool listings with the full (raw) capacity of the drives, while zfs listings will show the space minus parity...

But what you've shown above is:

A 9-disk RAIDZ1 striped with a 3-disk RAIDZ1 and a 6-disk RAIDZ1.

That's pretty bad if you weren't intending to do that. Are all of the drives the same size?