Analyzing Linux NFS server performance
I'd like to do some analysis of our NFS server to help track down potential bottlenecks in our applications. The server is running SUSE Enterprise Linux 10.
The kind of things I'm looking to know are:
- Which files are being accessed by which clients
- Read/write throughput on a per-client basis
- Overhead imposed by other RPC calls
- Time spent waiting on other NFS requests, or disk I/O, to service a client
I already know about the statistics available in /proc/net/rpc/nfsd and in fact I wrote a blog post describing them in depth. What I'm looking for is a way to dig deeper and help understand what factors are contributing to the performance seen by a particular client. I want to analyze the role the NFS server plays in the performance of an application on our cluster so that I can think of ways to best optimize it.
Just an idea, try to sniff nfs traffic with wireshark. Might tell you which user accessed what file:
tshark -R nfs -i eth0
I have to say of all the different *stat utilities available to one, nfsstat is by far the worst! It gives you the ability to look at a bunch of counters but that's all. If you look at them twice, YOU have to do the work of trying to figure out by how much each counter changed and if you want to know the rate of change you then need to divide by the number of seconds between samples. In all fairness, nfsstat does date back many years when things were still pretty crude and is now hampered by nobody wanting to change the output format because it would probably break a lot of things.
As for using collectl to monitor nfs, it does provide nfsstat output in a much easier to read format, but what's even better you can let it run for hours or days and play back the data you had collected in the background. As for the request to see what processes are doing, collectl can also gather process data including how much I/O each process is doing and even play it back showing the top I/O users. You can also use the top feature in real time.
If you want to watch the disks theme selves collectl can do that too and display everything in a coordinated display.
Check it out... -mark
collectl (especially its NFS subsystem) is a very nice utility which might be useful for your analysis but it does not match your requirements list. I'm not aware of any Linux utility that does.
(Please let me add this off-topic note: There is software which matches your requirements: Sun's DTrace-based Analytics (pdf) - but unfortunately is not available on Linux. You'll find lots of great examples in Brendan Gregg's blog which illustrate the capabilities of this tool.)
Check out nfsstat. It doesn't show everything you want but at least a good subset.
http://linux.die.net/man/8/nfsstat