Storing many small files with xattr in XFS and ext4

One thing is that XFS doesn't perform well on millions of small sizes. XFS is not inode based FS. Your block sizes will differ as you have lots of small files, ext4 holds in data blocks group also indirect block maps, extent tree blocks, and extended attributes, inode lists e.t.c. which increase total block size for the same amount of files. Also journaling takes some. For ext4 you could remove journaling if you want, but recovery of fs time will increase significantly, also you could allocate less % for superuser using -m option (default is 5%) you could put up 1%.