FreeBSD ZFS RAID-Z2 performance issues

I suspect you are probably running into the issue that I'm going to call the "512 byte sectors are not 4K sectors" issue. Anyway, google up on gnop, 4K sector, WD Green and I suspect you'll find the fix. I first learned of it on this site, and it was very informative about that issue, as well as a variety of other tunings for FreeBSD and ZFS. Good luck!

Edit: to quote from the linked site:

... Finally, I came across references to problems with Western Digital’s 1.5 TB (WE15EADS) Green drives that I am using.

The drives have a 4KB physical sector but report 512 Bytes to the BIOS. So performance drops off on really big writes because zfs on FreeBSD sends 4KB of data to the drive as 8 separate writes of 512 bytes, which requires the firmware in the drive to increase its work load by an estimated factor of 60 (1st 512 Bytes - write 4KB, 2nd 512 Bytes, read 4K, write 4K, ..., 8th 512 Bytes, read 4K, write 4K -- so 4KB of writes become 4KB write + (4KB read + 4KB write)X(4KB/512Bytes - 1) = 60. The drives built in 32 MB cache helps until it fills and the zfs arc kicks in and then the arc begins to fill. So all in all, no big deal right?

Actually it is a very big deal if you are writing files to zfs that are larger than your arc plus the size of the buffer on the drive. ...

In addition to the 4K/512b issue, WD green disks are not the ideal choice for your use as they are parking their heads after 8 seconds of inactivity and it takes a couple of second for them to recover to a functional state. This is the write delay you are observing.

http://opensolaris.org/jive/thread.jspa?messageID=480641&tstart=0