10G issues centos6.7 juniper ex4500
I've been scratching my head over this for the past 1 week and can't seem to find any solution. So, this my problem... I have a Juniper ex4500 10g switch connected to my servers over sfp+ twinax cables.
1st server (filer): Dell R620
RAM : 96GB
N/W card : Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 58) (2 ports) on bond0 which makes this 20Gbps
OS : centos 6.7 x86_64
Storage : 24 disks connected via perc 810 on RAID 5
2st server (compute): Dell R810
RAM : 256GB
N/W card : Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 58) (2 ports) no bond
OS : centos 6.7 x86_64
Storage : single disk 512GB 6Bb/s
3rd server (compute): Dell R810
RAM : 512GB
N/W card : Ethernet controller: QLogic Corp. cLOM8214 1/10GbE Controller (rev 58) (2 ports) no bond
OS : centos 6.7 x86_64
Storage : single disk 512GB 6Bb/s
I have a CentOS 6.7 ISO as the file that I'm using for testing the transfer speed. And this is how I started my tests. First I started with server 2 and server 3 by using scp centos 6.7.iso root@server3:~/ and the speed it is transferring is maxing out to 167MB/sec. I figured it was because the MTU is 1500 and I'm getting lower speeds. So changed the MTU on the switch to 9214 on the ports the servers are connected to and in the servers ifcfg-px files and rebooted the servers. But now the transfer speed increased only to 200MB/sec. I tested with server 1 which has a bond on it and MTU set to 9000 I still get 200MB/s.
iperf says 10g as the bandwidth but I can't see it. Does scp have so much of an overhead that it is maxing out to 200MB/s?
Anyways, if anyone of you have any idea what the problem is please let me know. it is highly appreciated.
Below are my outputs of iperf and scp.
Client connecting to teng.tofu, TCP port 5001
TCP window size: 92.6 KByte (default)
[ 3] local 10.10.113.15 port 58221 connected with 10.10.113.21 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.5 GBytes 9.90 Gbits/sec
SCP
100% 3926MB 186.9MB/s 00:21
The limitation is most certainly the hard coded buffers in SSH and the additional hard coded buffers in the SCP client. You can get around that with the HPN ssh patch that ewwhite mentioned, but I would first consider other methods rather than modifying your ssh daemon. Or, make a second custom ssh daemon if you must.
If your traffic needs to be encrypted, there are several ways to set up a VPN between the hosts, then you can simply use traditional methods such as native rsync (rsync host::resource/path vs. rsync host:/path) or NFS. Native rsync is easier to secure if you are using an ad-hoc vpn setup such as OpenVPN, Tinc or the native IPSEC built into the kernel.
To see that the network is not likely your issue, create an NFS share, or fire up rsync as a daemon and use the native rsync protocol (not using ssh). To reach 10gb/s you will need to tune the buffer sizes in /etc/sysctl.conf (not there by default) and likely may need to adjust the default options of your gig-e drivers via /etc/modprobe.d/driver-name.conf that you would need to read up on.
Edit
If you are just looking to test your 10Gb cards, then fire up iperf3 as a server on one machine and use iperf3 as a client on the other machine. You should be able to find this in the EPEL repo if using centos/redhat or you can compile it yourself. iperf3 will show you the maximum speed your servers can potentially get not counting application overhead, then it will be up to you to tune sysctl.conf, gig module driver options and such to get higher speeds if you need it.