VMXNET3 performance on Linux on ESX 5.0

Solution 1:

That's a really old version of VMware ESXi. The performance does seem to get better with newer revisions, but what you are seeing is probably accurate for that vintage.

Is there any reason your VMware environment hasn't been patched or updated? VMware tools are usually tied to the host version and patches, so you're running 2012-era VMware tools on a 2012-vintage ESXi installation. I have tools version 9.4.0.

You can update the tools independently of the ESXi installation by leveraging the VMware Operating System-Specific Packages (OSPs).

See: How do I make sure VMware Tools is installed and running?

On a modern ESXi 5.5 local copy (same host) between vmxnet3 VMs running EL6.5, I see:

Client:

[root@xt ~]# iperf -t 30 -c mccartney
------------------------------------------------------------
Client connecting to mccartney_mirror, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local 172.16.2.12 port 35543 connected with 172.16.2.159 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  90.1 GBytes  25.8 Gbits/sec

Server:

[root@McCartney ~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 172.16.2.159 port 5001 connected with 172.16.2.12 port 35543
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  90.1 GBytes  25.8 Gbits/sec

Solution 2:

Try changing ethernet driver settings.

/sbin/ethtool --offload eth0 gso off tso off sg off gro off 

See

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/ch10s04.html

If you experience low performance with the para-virtualized network drivers, verify the setting for the GSO and TSO features on the host system. The para-virtualized network drivers require that the GSO and TSO options are disabled for optimal performance."

This is about virtio drivers, but I think the same might apply to vmx drivers.

You can also find a lot of similar articles about VMWare: https://info.pleasantsolutions.com/Blog/VMware_Networking_Speed_Issue