Why would ping give me fast rtt values, but take a long time for each response to come back?

I've got a bit of a basic networking question here regarding ping.

When pinging a particular host on a completely different subnet, I get a response like this:

PING myhost.example.com (1.2.3.4) 56(84) bytes of data.
64 bytes from 1.2.3.4: icmp_req=1 ttl=115 time=1.88 ms
64 bytes from 1.2.3.4: icmp_req=2 ttl=115 time=1.66 ms
64 bytes from 1.2.3.4: icmp_req=3 ttl=115 time=1.96 ms
64 bytes from 1.2.3.4: icmp_req=4 ttl=115 time=1.95 ms

--- myhost.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 48191ms
rtt min/avg/max/mdev = 1.669/1.868/1.969/0.127 ms

This initially looks normal, but then you see that the total time is 48 seconds to get these 4 responses. When the ping is running, there is a noticable gap of around 10 seconds before each line is printed. However, the rtt is pretty much what I'd expect.

This is causing a bit of an issue, as I'm trying to monitor this host with nagios, and the host check is currently getting a 'Network unreachable' error. The network is reachable, the rtt is sane, but I can't help wonder if this slow total time might have something to do with it.

The host is 14 hops away, and the traceroute looks like this (I've anonymised the intermediate steps, they are all different IPs!):

traceroute to myhost (1.2.3.4), 30 hops max, 60 byte packets
 1  10.A.B.C (10.A.B.C)  0.680 ms  0.729 ms  0.833 ms
 2  10.A.B.C (10.A.B.C)  0.636 ms  0.678 ms  0.759 ms
 3  10.A.B.C (10.A.B.C)  0.803 ms  0.867 ms  0.923 ms
 4  10.A.B.C (10.A.B.C)  0.702 ms  0.738 ms  0.782 ms
 5  10.A.B.C (10.A.B.C)  1.027 ms  1.251 ms  1.342 ms
 6  10.A.B.C (10.A.B.C)  2.688 ms  1.436 ms  1.484 ms
 7  10.A.B.C (10.A.B.C)  2.960 ms  3.475 ms  3.527 ms
 8  10.A.B.C (10.A.B.C)  1.284 ms  1.310 ms  1.393 ms
 9  10.A.B.C (10.A.B.C)  1.990 ms  1.865 ms  1.964 ms
10  10.A.B.C (10.A.B.C)  1.750 ms  1.841 ms  1.748 ms
11  10.A.B.C (10.A.B.C)  1.849 ms  1.614 ms  1.628 ms
12  10.A.B.C (10.A.B.C)  1.997 ms  2.150 ms  2.119 ms
13  10.A.B.C (10.A.B.C)  2.442 ms  2.454 ms  2.560 ms
14  1.2.3.4 (1.2.3.4)  1.978 ms * *

What would cause this?


This is a bit of a wild guess....

You are pinging by DNS name.
Does the DNS loookup of myhost.example.com take that long to resolve ?
What happens if you ping by ip-address ?


Try to edit /etc/nsswitch.conf. mDNS caused me the same problem.

If you have:

hosts:  files mdns4_minimal [NOTFOUND=return] dns mdns4

try to replace with:

hosts:  files dns

In addition to the DNS lookuping issue, it may also caused by the interval between ping packets:

-i interval
          Wait  interval  seconds  between  sending each packet.  The default is to wait for one second between each packet normally, or not to wait in
          flood mode. Only super-user may set interval to values less 0.2 seconds.

If you ping google's server by ip:

# ping -c 3 74.125.130.101
PING 74.125.130.101 (74.125.130.101) 56(84) bytes of data.
64 bytes from 74.125.130.101: icmp_seq=1 ttl=50 time=2.51 ms
64 bytes from 74.125.130.101: icmp_seq=2 ttl=50 time=2.52 ms
64 bytes from 74.125.130.101: icmp_seq=3 ttl=50 time=2.72 ms

--- 74.125.130.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2006ms
rtt min/avg/max/mdev = 2.511/2.589/2.729/0.114 ms

Total time is still much more larger than the sum of rtts.

But if you use ping in flood mode:

# ping -c 3 -f 74.125.130.101
PING 74.125.130.101 (74.125.130.101) 56(84) bytes of data.

--- 74.125.130.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 8ms
rtt min/avg/max/mdev = 2.474/2.744/3.140/0.286 ms, ipg/ewma 4.230/2.657 ms

or setting interval to 0:

# ping -c 3 -i 0 74.125.130.101
PING 74.125.130.101 (74.125.130.101) 56(84) bytes of data.
64 bytes from 74.125.130.101: icmp_seq=1 ttl=50 time=2.83 ms
64 bytes from 74.125.130.101: icmp_seq=2 ttl=50 time=2.87 ms
64 bytes from 74.125.130.101: icmp_seq=3 ttl=50 time=2.96 ms

--- 74.125.130.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 9ms
rtt min/avg/max/mdev = 2.834/2.891/2.966/0.083 ms, ipg/ewma 4.510/2.855 ms

Total time is nearly the sum of rtts.