Time Machine Sparse Bundle failed to verify after catalog rebuild

After reading about the fsck_hfs man page, I feel like the need to rebuild catalogs. However, when I do a repair before the rebuild, it succeeded, but then printed a lot of errors during the build, and then none again during an verify after the rebuild. Here's an output of rebuilding a local disk (Seagate Backup 5T)

Last login: Thu Aug 15 17:23:10 on ttys010
Welcome to fish, the friendly interactive shell
root@Joys-MBP /p/v/root# fsck_hfs -c 4294967296 -d -D 
fsck_hfs: option requires an argument -- D
usage: fsck_hfs [-b [size] B [path] c [size] e [mode] ESdfglx m [mode] npqruy] special-device
  b size = size of physical blocks (in bytes) for -B option
  B path = file containing physical block numbers to map to paths
  c size = cache size (ex. 512m, 1g)
  e mode = emulate 'embedded' or 'desktop'
  E = exit on first major error
  d = output debugging info
  f = force fsck even if clean (preen only) 
  g = GUI output mode
  x = XML output mode
  l = live fsck (lock down and test-only)
  m arg = octal mode used when creating lost+found directory 
  n = assume a no response 
  p = just fix normal inconsistencies 
  q = quick check returns clean, dirty, or failure 
  r = rebuild catalog btree 
  S = Scan disk for bad blocks
  u = usage 
  y = assume a yes response 
root@Joys-MBP /p/v/root# fsck_hfs -c 4294967296 -d -D /dev/disk2
fsck_hfs: invalid debug development argument.  Assuming zero
fsck_hfs: missing special-device
usage: fsck_hfs [-b [size] B [path] c [size] e [mode] ESdfglx m [mode] npqruy] special-device
  b size = size of physical blocks (in bytes) for -B option
  B path = file containing physical block numbers to map to paths
  c size = cache size (ex. 512m, 1g)
  e mode = emulate 'embedded' or 'desktop'
  E = exit on first major error
  d = output debugging info
  f = force fsck even if clean (preen only) 
  g = GUI output mode
  x = XML output mode
  l = live fsck (lock down and test-only)
  m arg = octal mode used when creating lost+found directory 
  n = assume a no response 
  p = just fix normal inconsistencies 
  q = quick check returns clean, dirty, or failure 
  r = rebuild catalog btree 
  S = Scan disk for bad blocks
  u = usage 
  y = assume a yes response 
root@Joys-MBP /p/v/root# fsck_hfs -c 4294967296 -d -D /dev/disk2s2
fsck_hfs: invalid debug development argument.  Assuming zero
fsck_hfs: missing special-device
usage: fsck_hfs [-b [size] B [path] c [size] e [mode] ESdfglx m [mode] npqruy] special-device
  b size = size of physical blocks (in bytes) for -B option
  B path = file containing physical block numbers to map to paths
  c size = cache size (ex. 512m, 1g)
  e mode = emulate 'embedded' or 'desktop'
  E = exit on first major error
  d = output debugging info
  f = force fsck even if clean (preen only) 
  g = GUI output mode
  x = XML output mode
  l = live fsck (lock down and test-only)
  m arg = octal mode used when creating lost+found directory 
  n = assume a no response 
  p = just fix normal inconsistencies 
  q = quick check returns clean, dirty, or failure 
  r = rebuild catalog btree 
  S = Scan disk for bad blocks
  u = usage 
  y = assume a yes response 
root@Joys-MBP /p/v/root# fsck_hfs -c 4294967296 -d -D 0x0001 /dev/disk2s2
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
fsck_hfs: Volume is journaled.  No checking performed.
fsck_hfs: Use the -f option to force checking.
    CheckHFS returned 0, fsmodified = 0
root@Joys-MBP /p/v/root# fsck_hfs -c 4294967296 -d -D 0x0002 /dev/disk2s2
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
fsck_hfs: Volume is journaled.  No checking performed.
fsck_hfs: Use the -f option to force checking.
    CheckHFS returned 0, fsmodified = 0
root@Joys-MBP /p/v/root# fsck_hfs -c 4294967296 -d -D 0x0002 0x0001 /dev/disk2s2

0x0001: No such file or directory
Can't stat 0x0001
Can't stat 0x0001: No such file or directory
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
fsck_hfs: Volume is journaled.  No checking performed.
fsck_hfs: Use the -f option to force checking.
    CheckHFS returned 0, fsmodified = 0
root@Joys-MBP /p/v/root# fsck_hfs -c 17592186044415M -d -D 0x0002 0x0001 0x0010 0x0020 /dev/disk2s2
0x0001: No such file or directory
Can't stat 0x0001
Can't stat 0x0001: No such file or directory
0x0010: No such file or directory
Can't stat 0x0010
Can't stat 0x0010: No such file or directory
0x0020: No such file or directory
Can't stat 0x0020
Can't stat 0x0020: No such file or directory
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
fsck_hfs: Volume is journaled.  No checking performed.
fsck_hfs: Use the -f option to force checking.
    CheckHFS returned 0, fsmodified = 0
root@Joys-MBP /p/v/root# fsck_hfs -c 17592186044415M -d -D 0x0002 -D 0x0001 -D 0x0010 -D 0x0020 /dev/disk2s2
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
fsck_hfs: Volume is journaled.  No checking performed.
fsck_hfs: Use the -f option to force checking.
    CheckHFS returned 0, fsmodified = 0
root@Joys-MBP /p/v/root# fsck_hfs -c 1759218604441M -d -D 0x0002 -D 0x0001 -D 0x0010 -D 0x0020 /dev/disk2s2
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
fsck_hfs: Volume is journaled.  No checking performed.
fsck_hfs: Use the -f option to force checking.
    CheckHFS returned 0, fsmodified = 0
root@Joys-MBP /p/v/root# fsck_hfs -c 4g -d -D 0x0002 -D 0x0001 -D 0x0010 -D 0x0020 /dev/disk2s2
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
fsck_hfs: Volume is journaled.  No checking performed.
fsck_hfs: Use the -f option to force checking.
    CheckHFS returned 0, fsmodified = 0
root@Joys-MBP /p/v/root# fsck_hfs -c 4g -d -D 0x0002 -D 0x0001 -D 0x0010 -D 0x0020 -f /dev/disk2s2
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
   The volume name is Seagate 5T (Joy Jin)
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
** Checking volume information.
** The volume Seagate 5T (Joy Jin) appears to be OK.
    CheckHFS returned 0, fsmodified = 0
root@Joys-MBP /p/v/root# fsck_hfs -c 4g -d -D 0x0002 -D 0x0001 -D 0x0010 -D 0x0020 -f -R ace /dev/disk2s2
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
   The volume name is Seagate 5T (Joy Jin)
** Checking extents overflow file.
** Checking catalog file.
** Rebuilding extents overflow B-tree.
hfs_UNswap_BTNode: invalid node height (1)
** Rebuilding catalog B-tree.
hfs_UNswap_BTNode: invalid node height (1)
** Rebuilding extended attributes B-tree.
hfs_UNswap_BTNode: invalid node height (1)
** Rechecking volume.
** Checking Journaled HFS Plus volume.
   The volume name is Seagate 5T (Joy Jin)
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
   Unused node is not erased (node = 6360)
   (message repeat from 6360 to 49663) --> I exceeded character limit (1916033/30000) so cut this part off
   Unused node is not erased (node = 49663)
** Checking volume bitmap.
** Checking volume information.
   Invalid volume file count
   (It should be 1143585 instead of 872171)
   Invalid volume directory count
   (It should be 201272 instead of 168319)
   Invalid volume free block count
   (It should be 310948865 instead of 350464208)
    invalid VHB nextCatalogID 
   Volume header needs minor repair
(2, 0)
   Verify Status: VIStat = 0x8000, ABTStat = 0x0004 EBTStat = 0x0000
                  CBTStat = 0x0000 CatStat = 0x00000000
** Repairing volume.
** Rechecking volume.
** Checking Journaled HFS Plus volume.
   The volume name is Seagate 5T (Joy Jin)
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
** Checking volume information.
** The volume Seagate 5T (Joy Jin) was repaired successfully.
    CheckHFS returned 0, fsmodified = 1
root@Joys-MBP /p/v/root# 

And then, I decided to do this to my Time Machine sparsebundle on Synology NAS (local 1gbps network). It failed. What should I do now?

Last login: Fri Aug 16 14:40:08 on ttys010
Welcome to fish, the friendly interactive shell
root@Joys-MacBook-Pro /p/v/root# fsck_hfs -df /dev/disk2s2
^C⏎                                                                             root@Joys-MacBook-Pro /p/v/root# 
date ; fsck_hfs -df /dev/disk2s2 ; date ; fsck_hfs -c 4g -d -D 0x0002 -D 0x0001 -D 0x0010 -D 0x0020 -f -R ace /dev/disk2s2 ; date
2019年 8月16日 星期五 14时40分44秒 CST
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Using cacheBlockSize=32K cacheTotalBlock=65536 cacheSize=2097152K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
** Detected a case-sensitive volume.
   The volume name is 时间机器备份
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking multi-linked directories.
    privdir_valence=17552, calc_dirlinks=104189, calc_dirinode=17552
** Checking volume bitmap.
** Checking volume information.
** The volume 时间机器备份 appears to be OK.
    CheckHFS returned 0, fsmodified = 0
2019年 8月16日 星期五 15时44分22秒 CST
journal_replay(/dev/disk2s2) returned 0
** /dev/rdisk2s2
    Cache size should be greater than 32M and less than 17592186044415M
    Using cacheBlockSize=32K cacheTotalBlock=98304 cacheSize=3145728K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
** Detected a case-sensitive volume.
   The volume name is 时间机器备份
** Checking extents overflow file.
** Checking catalog file.
** Rebuilding extents overflow B-tree.
hfs_UNswap_BTNode: invalid node height (1)
** Rebuilding catalog B-tree.
hfs_UNswap_BTNode: invalid node height (1)
** Rebuilding extended attributes B-tree.
hfs_UNswap_BTNode: invalid node height (1)
** Rechecking volume.
** Checking Journaled HFS Plus volume.
   Invalid extent entry
(4, 0)
    CheckExtRecord: id=4 1:(16384,0), (blockCount == 0)
** The volume   could not be verified completely.
    volume check failed with error 7 
    volume type is pure HFS+ 
    primary MDB is at block 0 0x00 
    alternate MDB is at block 0 0x00 
    primary VHB is at block 2 0x02 
    alternate VHB is at block 22476447294 0x53bb35e3e 
    sector size = 512 0x200 
    VolumeObject flags = 0x07 
    total sectors for volume = 22476447296 0x53bb35e40 
    total sectors for embedded volume = 0 0x00 
    CheckHFS returned -1317, fsmodified = 1
2019年 8月16日 星期五 16时12分03秒 CST
root@Joys-MacBook-Pro /p/v/root# fsck_hfs -df /dev/disk2s2
/dev/disk2s2: No such file or directory
Can't stat /dev/disk2s2
Can't stat /dev/disk2s2: No such file or directory
root@Joys-MacBook-Pro /p/v/root# fsck_hfs -df /dev/disk2s2
Unable to open block device /dev/disk2s2: Resource busyjournal_replay(/dev/disk2s2) returned 16
** /dev/rdisk2s2
    Using cacheBlockSize=32K cacheTotalBlock=65536 cacheSize=2097152K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
   Invalid extent entry
(4, 0)
    CheckExtRecord: id=4 1:(16384,0), (blockCount == 0)
** The volume   could not be verified completely.
    volume check failed with error 7 
    volume type is pure HFS+ 
    primary MDB is at block 0 0x00 
    alternate MDB is at block 0 0x00 
    primary VHB is at block 2 0x02 
    alternate VHB is at block 22476447294 0x53bb35e3e 
    sector size = 512 0x200 
    VolumeObject flags = 0x07 
    total sectors for volume = 22476447296 0x53bb35e40 
    total sectors for embedded volume = 0 0x00 
    CheckHFS returned -1317, fsmodified = 0
root@Joys-MacBook-Pro /p/v/root# fsck_hfs -df /dev/disk2s2
Unable to open block device /dev/disk2s2: Resource busyjournal_replay(/dev/disk2s2) returned 16
** /dev/rdisk2s2
    Using cacheBlockSize=32K cacheTotalBlock=65536 cacheSize=2097152K.
   Executing fsck_hfs (version hfs-407.50.6).
** Checking Journaled HFS Plus volume.
   Invalid extent entry
(4, 0)
    CheckExtRecord: id=4 1:(16384,0), (blockCount == 0)
** The volume   could not be verified completely.
    volume check failed with error 7 
    volume type is pure HFS+ 
    primary MDB is at block 0 0x00 
    alternate MDB is at block 0 0x00 
    primary VHB is at block 2 0x02 
    alternate VHB is at block 22476447294 0x53bb35e3e 
    sector size = 512 0x200 
    VolumeObject flags = 0x07 
    total sectors for volume = 22476447296 0x53bb35e40 
    total sectors for embedded volume = 0 0x00 
    CheckHFS returned -1317, fsmodified = 0
root@Joys-MacBook-Pro /p/v/root# 

Solution 1:

Well, basically I recovered from my NAS's version history, although that take a long time (and failed several times). Then I started creating snapshots regularly, and I can revert to a previous state whenever I messed up.

Another thing to note: before rebuilding the catalogs, first run a regular fsck. Usually if I do that I would end up with a successful rebuild, otherwise it will fail.

If you do not have a backup of any kind, you can preserve your history by creating a new sparse bundle, copy the backups.db folder over using Finder (first mounting the corrupt sparse bundle read-only), and backup to that new sparse bundle. Note that you may have to use tmutil inheritbackup and/or tmutil associatedisk to preserve the history.