Why isn't the arc_max setting honoured on ZFS on Linux?

Solution 1:

IMHO the parameters in /sys/module/zfs/parameters can only be set to 0 / 1 - disabled / enabled." Correction: depends on the parameter

I am in the same boat wanting to limit zfs' memory usage and it seems one has to create a /etc/modprobe.d/zfs.conf file and enter the parameter and the desired value in there. This change will take effect upon reboot.

echo "options zfs zfs_arc_max=34359738368" >> /etc/modprobe.d/zfs.conf

To effect the running module one can change the zfs_arc_max parameter.

echo "34359738368" > /sys/module/zfs/parameters/zfs_arc_max

Please note the use of > to replace the content of the file in contrast to adding to the file with >>.

source: https://stackoverflow.com/a/18808311

Solution 2:

Once you modify your arc size, you need to drop your caches.

echo 3 > /proc/sys/vm/drop_caches

and wait (your prompt won't return immediately, but other processes will continue to run). It will slowly unload the caches (2.5 min for my 24GB cache on 2 pair of raid 1'd 2TB WD blacks on a 2Ghz 4 year old cpu on a box w/64GB) - beware, you will suddenly have no caches, and any processes reading data will be pulling off raw disk so you'll probably see IO wait jump up for a while til cache is repopulated.