Gluster + ZFS, deadlock during benchmarking: zfs_iput_taskq 100% cpu

ZFS on Linux needs a bit of basic tuning in order to operate well under load. There's a bit of a struggle between the ZFS ARC and the Linux virtual memory subsystem.

For your CentOS systems, try the following:

Create an /etc/modprobe.d/zfs.conf configuration file. This is read during the module load/boot.

Add something like:

options zfs zfs_arc_max=40000000000
options zfs zfs_vdev_max_pending=24

Where zfs_arc_max is roughly 40% of your RAM in bytes (Edit: try zfs_arc_max=1200000000). The compiled-in default for zfs_vdev_max_pending is 8 or 10, depending on version. The value should be high (48) for SSD or low-latency drives. Maybe 12-24 for SAS. Otherwise, leave at default.

You'll want to also have some floor values in /etc/sysctl.conf

vm.swappiness = 10
vm.min_free_kbytes = 512000

Finally, with CentOS, you may want to install tuned and tuned-utils and set your profile to virtual-guest with tuned-adm profile virtual-guest.

Try these and see if the problem persists.

Edit:

Run zfs set xattr=sa storage. Here's why. You may have to wipe the volumes and start again (I'd definitely recommend doing so).