A truer gauge of network speed

I was recently setting up a new 1.5MB DSL connection at home. After connecting my computers (wirelessly) to my DSL modem I fired off a broadband speed tester to see if both were delivering the expected throughput. My WinXP box was delivering about 0.7MB while my WinVista box was delivered about 1.2MB. Clearly I had an issue localized to the WinXP box so I tweaked its network settings with TCP Optimizer. That made its speed comparable to my other machine, both at about 1.2MB.

Reflecting upon this I asked myself: how do I really know what my speed is? I decided to test further, applying 3 different speed testers using 2 different browsers on each of 2 machines and, for good measure, I repeated each test twice, yielding 24 data points (well, OK, 48 since there was upstream and downstream). The raw data appears below. Note that the data is sorted by provider then by speed (in Kbps) as the most useful order to see trends--they are not shown in the order of trials performed. Also note that I shutdown all other foreground apps and windows but did not meticulously check service activity so that could possibly account for some of the glitches.

Observations on downstream activity

• There are 3 outliers (2 from Speed.IO on the XP box only, and one on from AuditMyPC on Vista) and all 3 are on Firefox rather than IE8.
• The standard deviation is 244 when I include all data, but just 56 when I remove the 3 outliers.
• SpeedTest was remarkably consistent across browsers and OSes.
• Ignoring outliers, the averages per service are:
  • SpeedTest = 1334
  • Speed.IO = 1243
  • AuditMyPc = 1217

Observations on upstream activity

• There are no true outliers but the scatter is much larger than downstream.
• The standard deviation is 264 including all data.
• SpeedTest and Speed.IO were remarkably consistent across browsers and OSes, while AuditMyPc had a wild ride (though faster than SpeedTest or Speed.IO in all cases).
• The averages per service are:
  • SpeedTest = 428
  • Speed.IO = 468
  • AuditMyPc = 930

The question

I have some estimate of my connection speed (1270 Kpbs downstream, 600 Kpbs upstream). The downstream estimate is much more reliable based on standard deviations given above. My question is how does one get truer or more accurate estimates of connection speed? (Note I ask for truer rather than true--I am a pragmatist:-) Or to put it another way, how does one judge which numbers to believe?

The raw data

(2 machines x 2 browsers x 3 services x 2 trials x 2 directions)

Machine Browser Service     Direction   Speed
XP      IE8     SpeedTest   Downstream  1340
XP      IE8     SpeedTest   Downstream  1340
Vista   IE8     SpeedTest   Downstream  1340
XP      FireFox SpeedTest   Downstream  1330
Vista   FireFox SpeedTest   Downstream  1330
Vista   IE8     SpeedTest   Downstream  1330
XP      FireFox SpeedTest   Downstream  1330
Vista   FireFox SpeedTest   Downstream  1330
Vista   IE8     Speed.IO    Downstream  1264
Vista   IE8     Speed.IO    Downstream  1263
XP      IE8     Speed.IO    Downstream  1240
XP      IE8     Speed.IO    Downstream  1235
Vista   FireFox Speed.IO    Downstream  1231
Vista   FireFox Speed.IO    Downstream  1226
XP      FireFox Speed.IO    Downstream  630
XP      FireFox Speed.IO    Downstream  353
XP      FireFox AuditMyPc   Downstream  1242
XP      FireFox AuditMyPc   Downstream  1242
XP      IE8     AuditMyPc   Downstream  1240
Vista   IE8     AuditMyPc   Downstream  1208
Vista   IE8     AuditMyPc   Downstream  1202
Vista   FireFox AuditMyPc   Downstream  1200
XP      IE8     AuditMyPc   Downstream  1182
Vista   FireFox AuditMyPc   Downstream  780
XP      FireFox SpeedTest   Upstream    430
XP      IE8     SpeedTest   Upstream    430
Vista   FireFox SpeedTest   Upstream    430
Vista   IE8     SpeedTest   Upstream    430
XP      FireFox SpeedTest   Upstream    430
XP      IE8     SpeedTest   Upstream    430
Vista   IE8     SpeedTest   Upstream    430
Vista   FireFox SpeedTest   Upstream    410
Vista   FireFox Speed.IO    Upstream    473
Vista   FireFox Speed.IO    Upstream    472
Vista   IE8     Speed.IO    Upstream    471
XP      FireFox Speed.IO    Upstream    468
Vista   IE8     Speed.IO    Upstream    466
XP      FireFox Speed.IO    Upstream    465
XP      IE8     Speed.IO    Upstream    464
XP      IE8     Speed.IO    Upstream    461
Vista   IE8     AuditMyPc   Upstream    1191
XP      FireFox AuditMyPc   Upstream    1165
XP      IE8     AuditMyPc   Upstream    1074
XP      IE8     AuditMyPc   Upstream    1022
XP      FireFox AuditMyPc   Upstream    884
Vista   IE8     AuditMyPc   Upstream    851
Vista   FireFox AuditMyPc   Upstream    677
Vista   FireFox AuditMyPc   Upstream    574

Ciaran makes a good point in his comment; you have to know what the test is measuring, and this is not clear with a proprietary bandwidth tester.

Downloading a large file will give you a good idea of your TCP throughput, but this measurement will be very sensitive to your latency (Ping time) to the server, and may be affected by your TCP configuration settings (critically your TCP receive window).

Tools like IPerf and ixChariot are used by networking professionals , but these are not really suitable for casual use (IPerf does not work easily behind NAT for example). To gauge the ultimate capacity of your pipe you could flood your connection with a uni-directional stream of UDP datagrams using one of the above tools.

Online bandwidth tools use some tricks to try and push up the throughput beyond the performance of a single TCP connection. I find that speedtest.net generally gives you an average download speed ~30-40% higher than what you would measure by timing the download of a large file. I have confirmed this by timing downloads of the files used by speedtest itself.

Using Wireshark, I made the following observations;

1. The speedtest client downloads two files simaultaneously while measuring bandwidth - this gives a slightly higher aggregate throughput than a single file
2. The speedtest speedometer reports a high percentile (90-95%) of the per-second instantaneous throughputs - not the long term average

The speedtest GUI suggests that your download of an mp3 would proceed at the measured rate. This is slightly misleading as the measured rate is more representative of the peak throughput achievable on your link.

Another wrinkle that makes it even harder to find your 'true' speed...I know Comcast does this, not sure about other ISPs. Basically they give you 1.5-2x speed for the first 10-20MB of a download, before dropping you back down to your normal speed. Unfortunately most speed tests don't download more than 10MB, so it really messes with the results.

Personally I think the NDT speed testers are one of the most accurate ones, assuming they have a sever that is near to you. Another thing to add to your data points.