Can't mount nfs volume - time out
I have an NFSv3 export from a Linux fileserver which used to mount fine. The fileserver had to go down for hardware maintenance. After bringing the server back up, the Linux client can no longer mount the nfs export.
No configuration has changed on the server or the client. I did do a software update and reboot the client after the first mount failed, but that did not help.
[root@client ~]# showmount -e ark
Export list for ark:
/mnt/bigraid *
[root@client ~]# mount -t nfs ark:/mnt/bigraid raid
It just hangs at this point. In another terminal...
[root@client ~]# dmesg | tail
[ 2526.676437] nfs: server ark not responding, timed out
[ 2529.183107] nfs: server ark not responding, timed out
[ 2531.689778] nfs: server ark not responding, timed out
[ 2538.196432] nfs: server ark not responding, timed out
[ 2540.703107] nfs: server ark not responding, timed out
[ 2543.209767] nfs: server ark not responding, timed out
[ 2545.716436] nfs: server ark not responding, timed out
[ 2548.223098] nfs: server ark not responding, timed out
[ 2550.729775] nfs: server ark not responding, timed out
[ 2557.236435] nfs: server ark not responding, timed out
...OK, but I could see the exports with showmount...
[root@client ~]# ping ark
PING ark.homebase (10.10.10.2) 56(84) bytes of data.
64 bytes from ark.homebase (10.10.10.2): icmp_seq=1 ttl=64 time=0.067 ms
64 bytes from ark.homebase (10.10.10.2): icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from ark.homebase (10.10.10.2): icmp_seq=3 ttl=64 time=0.048 ms
64 bytes from ark.homebase (10.10.10.2): icmp_seq=4 ttl=64 time=0.042 ms
^C
--- ark.homebase ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
So I don't get it.
The server is running OpenSUSE. I ensured that the firewall was off (not that it was ever on) and that network connectivity seems fine.
ark:/etc # cat exports
/mnt/bigraid *(rw,root_squash,insecure,no_subtree_check,sync)
Edit: here's the list of RPC ports in use
ark:/etc/init.d # rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100005 1 udp 37599 mountd
100005 1 tcp 33880 mountd
100005 2 udp 37599 mountd
100005 2 tcp 33880 mountd
100005 3 udp 37599 mountd
100005 3 tcp 33880 mountd
100024 1 udp 49522 status
100024 1 tcp 41314 status
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100021 1 udp 51887 nlockmgr
100021 3 udp 51887 nlockmgr
100021 4 udp 51887 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 tcp 49804 nlockmgr
100021 3 tcp 49804 nlockmgr
100021 4 tcp 49804 nlockmgr
100000 2 udp 111 portmapper
Edit 2: Got some tcpdump information
(Edit 3: Removed tcpdump output, since it may not be relevant.)
I'm not at all familiar with what a proper nfs negotiation looks like. I also dumped a pcap file if you want to look at the data segments. It's at filedropper
Edit 3: I may be hitting this issue
I was following @CIA's advice below, and did this:
ark:/etc/init.d # ./nfsserver stop
Shutting down kernel based NFS server: nfsd statd mountd idmapd done
ark:/etc/init.d # ./portmap stop
Shutting down RPC portmap daemon done
ark:/etc/init.d # ./portmap start
Starting RPC portmap daemon done
ark:/etc/init.d # ./nfsserver start
Starting kernel based NFS server: idmapdexportfs: Warning: /mnt/bigraid does not support NFS export.
mountd statd nfsd sm-notify done
Despite the warning, the export now seems mountable.
Solution 1:
So, NFS is weird in the it relys on portmapper to be running, so it can map a specific port to an RPC port. (I guess it's not weird. It's just the way it works.) If NFS is up before portmapper, NFS doesn't know how to route requests, because it checks portmapper for this at the start of the process. If portmapper is not up before NFS, NFS doesn't know how to map port to rpc.
Here's more documentation about the process (even though its for CentOS, it's still relevant): http://www.centos.org/docs/5/html/Deployment_Guide-en-US/s2-nfs-methodology-portmap.html
As for your new error message, reboot the box you're mounting with and remount to see if the error comes back.
Solution 2:
to summarize the solution from given answers, following steps below directed me to the right direction to fixing NFS mount issue without restating the box.
-
run a tcpdump in client side to the NFS server IP address (assuming it's 1.2.3.4)
tcpdump -i <replace-with-correct-INTERFACE_name -n host 1.2.3.4 -
keep running tcpdump and try mounting the NFS share path.
-
lookup for the ports NFS is communicating with the server to mounting the path (in your case [tcpdump result in edit revision], shows those are ports: 880, 2049 only).
-
do telnet on NFS server IP and all the ports you got from tcpdump output in step3 and make sure you have telnet and ports are open from server side to client (for your case, only below two ports found).
telnet 1.2.3.4 880 telnet 1.2.3.4 2049 -
if you don't have telnet on any of those ports captured in step3, you need to open those ports at netwrok level (server side and/or firewall if you have)
-
try to mount the NFS again now.