How to run a command once a ZFS scrub *completes*?

I would like to use cron to schedule periodic scrubs of my ZFS pool, and at some reasonably short time after the scrub finishes, email a status report to myself. The purpose of this is to catch any problems without having to manually look for them (push rather than pull).

The first part is easy: just set up a cron job to run zpool scrub $POOL as root at whatever interval is reasonable in my particular situation.

The second part, I'm not quite so sure how to do. zpool scrub returns immediately and then the scrub is run in the background by the system (which is certainly desirable behavior if the scrub is initiated by an administrator from a terminal). zpool status gives me a status report and exits (with exit code 0 while the scrub is running; it hasn't finished yet so I don't know if the exit status changes once it's done, but I doubt it). The only parameter documented for zpool scrub is -s for "stop scrubbing".

The main problem is detecting the change of status from scrubbing to finished scrubbing. Given that, the rest should fall into place.

Ideally, I'd want to tell zpool scrub to not return until the scrub finishes, but I don't see any way to make it do that. (It would make it almost too easy to simply cron zpool scrub --wait-until-done $POOL; zpool status $POOL.)

Failing that, I'd like to ask the system whether a scrub is currently in progress, preferably in a way that doesn't too much risk breaking with an upgrade or configuration change, so that I can act on whether or not a previously running scrub has finished (by executing a zpool status when the scrub status goes from scrubbing to not scrubbing).

This particular setup is for a workstation system, so while a monitoring tool such as Nagios probably has add-ins that would solve the problem, it feels rather overkill to install such a tool for just this one task. Can someone suggest a lower-tech solution to the problem?

On ZFS On Linux, starting with version 0.6.3 this can be handled quite elegantly by using the ZFS Event Daemon (zed). The event daemon, by virtue of monitoring the kernel events directly, can react almost immediately to any events that take place and does not depend on continuous polling and parsing of some other command's output.

Create a shell script with any file name that begins with /etc/zfs/zed.d/scrub.finish (for example, scrub.finish-custom.sh). That script can take any appropriate action, such as sending an email, writing a log entry somewhere, or making the system sing and dance (OK, maybe not that). Examples are provided that can provide a starting point.

If all you want is to receive an email when the scrub completes, the provided scrub.finish-email.sh script will do that nicely. Simply edit /etc/zfs/zed.d/zed.rc to indicate to where the email should be sent and whether an email should be sent also if the pool is not experiencing any problems, make sure something named scrub.finish followed by anything in /etc/zfs/zed.d leads to it, and make sure zed is started on boot.

I use this simple script for scrubbing status reporting by email.

• zpadmin

If you need to detect transition from scrub running to scrub finished I would check the state field of zpool status output. Something like this:

# start scrubbing 
zpool scrub ZPOOL

# wait till scrub is finished
while zpool status ZPOOL | grep 'scan:  *scrub in progress' > /dev/null; do
   echo -n '.'
   sleep 10
done

# send a report
zpool status | mail -s "zpool status: ZPOOL" RECIPIENT

Although this question is specific to linux, it is the first google result when searching for "wait until scrub is finished", therefore I'd like to add some useful information for people running OpenSolaris (tested it on OmniOS, but SmartOS, illumos etc. should be similar) instead of Linux (normal Solaris should also work, but I did not test it there).

You can use syseventadm to register kernel events. The complete list can be found in /usr/include/sys/sysevent/eventdefs.h (just search for "ZFS" in this file). After adding events, the service has to be restarted, for example:

syseventadm add -c EC_zfs -s ESC_ZFS_scrub_finish /path/to/script.sh \$pool_name
syseventadm restart

This way, the script will be started when any scrubbing of any pool finishes - you have to check inside the script if $1 equals your desired pool name. Still, it is much less overhead than polling.

I have had much success with zfswatcher