How can I improve NFS performance on a gigabit LAN?

A couple of things:

To figure out whether the switch is the problem, take the switch out of the equation and test. Connect the machines back to back and test again. GigE cards will auto cross connect for you, no special cable needed.
After that, NFS tuning is a bit of a black art. From what I have seen, it tends to be very specific to the NFS server and sometimes the NFS client. Luckily, Google seems to have many papers on the subject. Read a bunch.

Often the ZFS Intent Log (ZIL) is the NFS bottleneck of a ZFS file server. ZFS obeys the NFS protocol very strictly and makes sure that the data of synchronous writes really hits the disk before it reports success to the writer. ZFS uses the ZIL to temporarily store this data before it writes it in a bigger chunk to the pool. If writes to your ZIL are slow (high latency) then NFS performance probably will be slow, too. A good test case of this problem is to extract a large tarball (many files) to the NFS export.

There are at least two solutions to improve the NFS performance:

Use a dedicated, fast storage device for the ZIL (think NVRAM or SSD). You can even create a ZIL mirror out of two devices. Warning: AFAIR it's not possible to remove a ZIL device after you've added it to a pool. Think twice. If you want to try SSDs for you ZIL you may want to read this interesting blog post: Somewhat stable Solid State
Disable the ZIL on your ZFS file server. Well, this of course breaks the protocol promise of synchronous writes at the risk of data integrity but it'll be much faster. Read this part of the ZFS Evil Tuning Guide before your try this alternative and make sure you understand the consequences.

A couple of hints:

try mounting the server via SMB/CIFS and seeing what kind of performance you get
try doing a test using ttcp or iperf to see what your network bandwidth is like from client to server

OS X NFS is not fantastic in our experience (OS X clients, Linux servers). We get better performance by using samba. YMMV!

Apache and linux file permissions

If $U$ is a real measurable function s.t. $\int_{B_1(x)}\frac{1}{U(y)} dy\to 0$, does $\int_{B_1(x)}\frac{1}{U(y)+a} dy\to 0$, too?

Baby Rudin theorem 10.20

What needed to be checked for a topological subgroup?

Relationship between angle of vectors and orthogonalization

Arzelà–Ascoli $\implies$ Dini's theorem

Why does the graph of $\sqrt[3]{x}$ look like that on GeoGebra although $x^3$ is a one-to-one function

$\lim_{n\to +\infty}\left(\frac{\ln x_n}{\sqrt[n]{e}-1}-n\right)$ where $x_n=\sum_{k=0}^n \frac{1}{k!}$ [closed]

Chern number of a product of complex projective lines

Let $f \in L^1(\mathbb{R})$ and $p$ a polynomial of deg $m$. Does $f / p \in L^1$ imply $x^m f / p \in L^1$ or vice versa?

Continuous Injective Map $f :\mathbb{S}^2 \rightarrow \mathbb{S}^1$