What are "good" speeds for iSCSI and NFS over 1GB ethernet?

I have an OpenSolaris box sharing out two ZFS filesystems. One is an NFS connection to a CentOS box running VMware server (the disk images are stored in ZFS). The other is an iSCSI connection to a Windows 2008 server with NTFS formatting on top of the ZFS. Both connections are direct over gig-E (no switches).

I'm running munin to monitor the boxes, but I'm not sure what kind of numbers I should be expecting. Can anybody give me some baseline numbers to compare against or make any suggestions on where to start tuning?

Here are the NFS stats I'm seeing, I'll post iSCSI once I fix munin on the solaris box :P

We've pushed a Sun X4100 writing over bonded GigE and iSCSI to an Sun X4500 to 280MB/s.

There's a lot that can be done to tune the TCP stack on Solaris to help things out, this my my stock tuning config. (take from some collection of Sun whitepapers):

$ cat /etc/rc3.d/S99ndd
#!/bin/bash

NDD=/usr/sbin/ndd

$NDD -set /dev/tcp tcp_xmit_hiwat 1048576
$NDD -set /dev/tcp tcp_recv_hiwat 8388608
$NDD -set /dev/tcp tcp_max_buf 8388608
$NDD -set /dev/udp udp_xmit_hiwat 1048576
$NDD -set /dev/udp udp_recv_hiwat 8388608
$NDD -set /dev/udp udp_max_buf 8388608
$NDD -set /dev/tcp tcp_conn_req_max_q 65536
$NDD -set /dev/tcp tcp_conn_req_max_q0 65536
$NDD -set /dev/tcp tcp_fin_wait_2_flush_interval 67500

Also, worth looking into on your OpenSolaris machine is changing the fsflush interval, the interrupt adjustment "magic" and disabling soft rings. Append the following to /etc/system (reboot required):

* "fsflush" tuning
set tune_t_fsflushr = 5
set autoup = 300
* Disable the Automatic Interrupt Adjustment
set dld:dld_opt = 2
* Disable "soft rings"
set ip:ip_squeue_fanout = 0
set ip:ip_soft_rings_cnt = 0

Worth mentioning, I do this on Solaris 10 -- not OpenSolaris -- but I think the tunables should work for you just the same.

I'm a big fan of Filebench for playing around with tuning options and doing throughput tests.

• FileBench on SolarisInternals Wiki
• FileBench on SourceForge

The (recently renamed) OpenSolaris 2009.06 release looks to be very exciting in the world of iSCSI and ZFS.

Hope this helps some!

I get around 90 MB/sec to my EMC AX150i arrays over iSCSI on 1GB ethernet.

For just a single dd or bonnie++ (raw speed, linear writing) you should get pretty close to the wire-speed.

But your bottleneck will be the disk array, once you start getting the random IO load of multiple VMs going, much more than the transport.

Also, if you don't have a battery backed write cache with a significant amount of RAM, your performance will crater as soon as you start getting a lot of writes with any other IO going on.