Why does 1GBit card have output limited to 80 MiB?
Solution 1:
80 MB / second is actually pretty good! That's about 640mbps, which is pretty darn close to the gigabit capacity of the NIC. If you take into consideration the TCPIP overhead, and disk speed you're probably at your maximum speed.
Solution 2:
Try putting this to your /etc/sysctl.conf
# General 10gigabit/LFP tuning
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_orphans=1048576
net.ipv4.tcp_orphan_retries=2
# Removes some internal buffering
net.ipv4.tcp_low_latency=1
# Time-wait sockets
# Do not turn on unless you know what you are doing!
#net.ipv4.tcp_tw_recycle=1
#net.ipv4.tcp_tw_reuse=1
# If PMTUD ICMP blackhole appears use
# RFC 4821, Packetization Layer Path MTU Discovery
net.ipv4.tcp_mtu_probing=1
# Netfilter's conntrack
# NB! For high-performance concerns you probably don't want to use `--state` rules at all
#net.ipv4.netfilter.ip_conntrack_max=1048576
#net.nf_conntrack_max=1048576
# SACKs are an optimization to TCP which in normal scenarios improves considerably performance.
# In Gigabit networks with no traffic competition these have the opposite effect.
# To improve performance they should be turned off with:
#net.ipv4.tcp_sack=0
# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout=15
# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time=1800
# Increased backlog (default: 100/1000 depending on kernel)
net.core.netdev_max_backlog=10000
net.core.somaxconn=10000
# Timestamps adds additional 12 bytes to header and uses CPU
# NB! It caused massive problems for me under benchmark load
# with a high count of concurrent connections.
# ( http://redmine.lighttpd.net/wiki/1/Docs:Performance )
#net.ipv4.tcp_timestamps=0
# Portrange for outgoing connections
# (increase the ephemeral port range)
# NB! After that tuning you probably do not want to listen on port >= 1024
net.ipv4.ip_local_port_range=1024 65535
# Fixing 'Too many open files', Second useful on nginx+aio workloads
fs.file-max=16777216
fs.aio-max-nr=65536
# If you are under DDoS you can
kernel.panic=10
# Lower following values
#net.ipv4.tcp_synack_retries=2
#net.ipv4.tcp_syn_retries=2
#net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait=15
#net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait=15
# If you under ping flood
#net.ipv4.icmp_echo_ignore_all=1
Each connection we make requires an ephemeral port, and thus a file descriptor, and by default this is limited to 1024. To avoid the Too many open files problem you’ll need to modify the ulimit for your shell. This can be changed in /etc/security/limits.conf
, but requires a logout/login. For now you can just sudo and modify the current shell (su back to your non-priv’ed user after calling ulimit if you don’t want to run as root):
ulimit -n 999999
Another thing you can try that may help increase TCP throughput is to increase the size of the interface queue. To do this, do the following:
ifconfig eth0 txqueuelen 1000
You can play with congestion control:
sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control=htcp
There is also some low level tuning, e.g. kernel module parameters
# /sbin/modinfo e1000
..snip...
parm: TxDescriptors:Number of transmit descriptors (array of int)
parm: TxDescPower:Binary exponential size (2^X) of each transmit descriptor (array of int)
parm: RxDescriptors:Number of receive descriptors (array of int)
parm: Speed:Speed setting (array of int)
parm: Duplex:Duplex setting (array of int)
parm: AutoNeg:Advertised auto-negotiation setting (array of int)
parm: FlowControl:Flow Control setting (array of int)
parm: XsumRX:Disable or enable Receive Checksum offload (array of int)
parm: TxIntDelay:Transmit Interrupt Delay (array of int)
parm: TxAbsIntDelay:Transmit Absolute Interrupt Delay (array of int)
parm: RxIntDelay:Receive Interrupt Delay (array of int)
parm: RxAbsIntDelay:Receive Absolute Interrupt Delay (array of int)
parm: InterruptThrottleRate:Interrupt Throttling Rate (array of int)
parm: SmartPowerDownEnable:Enable PHY smart power down (array of int)
parm: KumeranLockLoss:Enable Kumeran lock loss workaround (array of int)
parm: copybreak:Maximum size of packet that is copied to a new buffer on receive
And even lower level hardware tunings accessible via ethtool(1)
.
PS. Read kernel the docs, especially Documentation/networking/scaling.txt
PPS. While tuning TCP performance you may want to consult with RFC6349
PPPS. D-Link is not the best network hardware. Try Intel hardware with pci-x or pci-64