Strange Problem: Connection Reset By Peer
I am having some trouble with SSH on my linux server running CentOS. I can connect to my server fine using PuTTY or ssh from windows cmd. The same goes for using secure FTP. I can connect to the server, get a list of files and everything is okay. The problem occurs when I try to send any quantity of data across the network.
Whenever I attempt to transfer anything beyond a certain threshold, the connection fails and I see a 'Connection reset by peer' message. I have a sql file that is about 3 MB in my home directory. If I attempt to FTP it, it will begin the transfer and die after about 48k has been transferred. Then it will initiate a new connection and transfer another 48k. If I use PuTTY and open a session I can connect and login fine. If i attempt to cat file.sql again, the connection is terminated and I get a 'Connection reset by peer' message. Going from my local workstation to the server it's the same situation. I have quite a bit of source code that I need to commit to my svn repository hosted on the server, but the same 'Connection reset by peer' message appears.
I know the problem is on my local workstation because I can use my wife's macbook and ssh to the server with no problems. I can ssh into a friend's linux box (using the same putty install) and sftp to my server from their and download the file, open another ssh session from his box to my server and cat the file. So, something is going on, but i'm not sure what. Does anyone have any ideas?
Update
I have been trying to figure this out some more, and it seems like there is a hard limit of the amount of data that I can transfer in a single ssh session. I hit it immediately if I do cat file.sql but I can also keep typing ls -l a consistent number of times and will also get the 'Connection reset by peer' message. I've tried:
- generating new ssh keys
- restarting my router
- restarting my computer
- restarting the remote server
I wrote out a tcpdump on the remote server, but I don't understand TCP at such a detailed level that it makes much sense to me. I turned on debugging in ssh and here is the portion of the log leading up to the connection being reset:
Jul 24 23:10:56 server sshd[4507]: debug1: permanently_set_uid: 500/503
Jul 24 23:10:56 server sshd[4507]: debug1: Entering interactive session for SSH2.
Jul 24 23:10:56 server sshd[4507]: debug1: server_init_dispatch_20
Jul 24 23:10:56 server sshd[4507]: debug1: server_input_channel_open: ctype session rchan 256 win 16384 max 16384
Jul 24 23:10:56 server sshd[4507]: debug1: input_session_request
Jul 24 23:10:56 server sshd[4507]: debug1: channel 0: new [server-session]
Jul 24 23:10:56 server sshd[4507]: debug1: session_new: init
Jul 24 23:10:56 server sshd[4507]: debug1: session_new: session 0
Jul 24 23:10:56 server sshd[4507]: debug1: session_open: channel 0
Jul 24 23:10:56 server sshd[4507]: debug1: session_open: session 0: link with channel 0
Jul 24 23:10:56 server sshd[4507]: debug1: server_input_channel_open: confirm session
Jul 24 23:10:56 server sshd[4507]: debug1: server_input_channel_req: channel 0 request pty-req reply 1
Jul 24 23:10:56 server sshd[4507]: debug1: session_by_channel: session 0 channel 0
Jul 24 23:10:56 server sshd[4507]: debug1: session_input_channel_req: session 0 req pty-req
Jul 24 23:10:56 server sshd[4507]: debug1: Allocating pty.
Jul 24 23:10:56 server sshd[4505]: debug1: session_new: init
Jul 24 23:10:56 server sshd[4505]: debug1: session_new: session 0
Jul 24 23:10:56 server sshd[4507]: debug1: session_pty_req: session 0 alloc /dev/pts/2
Jul 24 23:10:56 server sshd[4507]: debug1: server_input_channel_req: channel 0 request shell reply 1
Jul 24 23:10:56 server sshd[4507]: debug1: session_by_channel: session 0 channel 0
Jul 24 23:10:56 server sshd[4507]: debug1: session_input_channel_req: session 0 req shell
Jul 24 23:10:56 server sshd[4508]: debug1: Setting controlling tty using TIOCSCTTY.
Jul 24 23:10:59 server sshd[4507]: Read error from remote host <my-ip>: Connection reset by peer
Jul 24 23:10:59 server sshd[4507]: debug1: do_cleanup
Jul 24 23:10:59 server sshd[4505]: debug1: do_cleanup
Jul 24 23:10:59 server sshd[4505]: debug1: PAM: cleanup
Jul 24 23:10:59 server sshd[4505]: debug1: PAM: deleting credentials
Jul 24 23:10:59 server sshd[4505]: debug1: PAM: closing session
Jul 24 23:10:59 server sshd[4505]: pam_unix(sshd:session): session closed for user <me>
Jul 24 23:10:59 server sshd[4505]: debug1: session_pty_cleanup: session 0 release /dev/pts/2
UPDATE 2:
About a week ago, I modified my ssh settings on my server using this wiki post: http://wiki.centos.org/HowTos/Network/SecuringSSH
Because I occasionally need to access my server from work, and because port 21 is open on our firewall, I changed the ssh port to 21. To further diagnose this problem, I tried to revert my ssh settings and changed the ssh port back to 22. Low and behold, the I do not encounter the error when I use port 22. Change it back to 21, and like clockwork when I hit 48k of data transferred - Connection reset by peer.
Given that I can get the initial connection and that I've had no trouble in the past establishing ftp connections on port 21, it doesn't seem like my firewall configuration is the issue.
At least at this point, I've got the problem narrowed down to the ssh port on my server. Flip it to 21 and instant problems, change it back to 22, no problem at all...
Can anyone think of why the listen port would make a difference? Again, it's only on my Windows XP box that it's causing problems. Let me know if anyone has any thoughts on what might cause this.
Update 2:
Just narrowed the problem down and I stand corrected - it is a firewall issue, but a Windows firewall issue, not at my router. If I use port 21 and disable the windows firewall I do not encounter the 'Connection reset by peer' message. To answer the obvious question, yes, port 21 is open on the windows firewall.
Since this computer is behind the firewall on my router, I can just disable it for now, but I would be interested to figure out what's going on here.
Solution 1:
You can solve using command line with this command (type this on the windows workstation as administrator):
netsh advfirewall set global statefulftp disable
Solution 2:
This could be related to your router trying to handle FTP NAT connection tracking automatically. It would only happen on port 21, not 22. Check out http://www.faqs.org/docs/iptables/complexprotocols.html