Right way of measuring overlay network performance

Solution 1:

UDP is both CPU and bandwidth bound. It sends packets without guaranteeing that they are sent, transmitted nor received.

• If the sender CPU is too busy, the packet is never sent.
• If the bandwidth can't keep up, packets are dropped in transit.
• If the receiver CPU is too busy or not ready to process incoming network data, it's lost.
• If the application doesn't extract packets from the OS (and process them) fast enough, they are lost.

Generally speaking, UDP performances are meaningless. Nothing prevents you to try to send 1 bazillion packets a second. That saturates the sender CPU and the network, while the receiver might not get much of anything.

If you really want to test UDP, that is a rather long topic that is worthy of a book. For starter, you need to monitor error rates and what data is actually sent/received.

You should test with TCP to measure the available bandwidth between hosts. iperf should be able to do that just fine.