ftplib storbinary with FTPS is hanging/never completing

I am trying to upload a file to an FTP site using FTPS, but when I attempt to store the file, it just hangs after the file is fully transferred.

global f_blocksize
global total_size
global size_written
f_blocksize = 1024
total_size = os.path.getsize(file_path)
size_written = 0
file = open(file_path, "rb")
try:
    ftps = FTP_TLS("ftp.example.com")
    ftps.auth()
    ftps.sendcmd("USER username")
    ftps.sendcmd("PASS password")
    ftps.prot_p()
    print(ftps.getwelcome())

    try:
        print("File transfer started...")
        ftps.storbinary("STOR myfile.txt", file, callback=handle, blocksize=f_blocksize)
        print("File transfer complete!")
    except OSError as ex:
        print(ftps.getresp())
except Exception as ex:
    print("FTP transfer failed.")
    print("%s: %s" %(type(ex), str(ex)))

def handle(block):
    global size_written
    global total_size
    global f_blocksize
    size_written = size_written + f_blocksize if size_written + f_blocksize < total_size else total_size
    percent_complete = size_written / total_size * 100
    print("%s percent complete" %str(percent_complete))

I get the following output:

220 Microsoft FTP Service
File transfer started...
3.5648389904264577 percent complete
7.129677980852915 percent complete
10.694516971279374 percent complete
14.25935596170583 percent complete
17.824194952132288 percent complete
21.389033942558747 percent complete
24.953872932985206 percent complete
28.51871192341166 percent complete
32.083550913838124 percent complete
35.648389904264576 percent complete
39.213228894691035 percent complete
42.778067885117494 percent complete
46.342906875543946 percent complete
49.90774586597041 percent complete
53.472584856396864 percent complete
57.03742384682332 percent complete
60.60226283724979 percent complete
64.16710182767625 percent complete
67.7319408181027 percent complete
71.29677980852915 percent complete
74.8616187989556 percent complete
78.42645778938207 percent complete
81.99129677980854 percent complete
85.55613577023499 percent complete
89.12097476066144 percent complete
92.68581375108789 percent complete
96.25065274151436 percent complete
99.81549173194082 percent complete
100.0 percent complete

After which there is no further progress until the connection times out...

FTP transfer failed.
<class 'ftplib.error_temp'>: 425 Data channel timed out due to not meeting the minimum bandwidth requirement.

While the program is running I can see an empty myfile.txt in the FTP site if I connect and look manually, but when I either cancel it or the connection times out, this empty file disappears.

Is there something I'm missing that I need to invoke to close the file after it has been completely transferred?

This appears to be an issue with Python's SSLSocket class, which is waiting for data from the server when running unwrap. Since it never receives this data from the server, it is unable to unwrap SSL from the socket and therefore times out.

This server in particular I have identified by the welcome message as some Microsoft FTP server, which fits in well with the issue written about in this blog

The "fix" (if you can call it that) was to stop the SSLSocket from attempting to unwrap the connection altogether by editing ftplib.py and amending the FTP_TLS.storbinary() method.

def storbinary(self, cmd, fp, blocksize=8192, callback=None, rest=None):
  self.voidcmd('TYPE I')
  with self.transfercmd(cmd, rest) as conn:
    while 1:
      buf = fp.read(blocksize)
      if not buf: break
      conn.sendall(buf)
      if callback: callback(buf)
    # shutdown ssl layer
    if isinstance(conn, ssl.SSLSocket):
      # HACK: Instead of attempting unwrap the connection, pass here
      pass
  return self.voidresp()

I faced this issue on STORBINARY function when using python's ftplib.FTP_TLS, prot_p and Microsoft FTP server.

Example:

ftps = FTP_TLS(host,username,password)
ftps.prot_p
STORBINARY...

The error indicated timeout on the unwrap function.

It is related to the following issues:

https://www.sami-lehtinen.net/blog/python-32-ms-ftps-ssl-tls-lockup-fix

https://bugs.python.org/issue10808

https://bugs.python.org/issue34557

Resolution:

1. Open the python page for ftplib: https://docs.python.org/3/library/ftplib.html

2. Click on the source code which will take you to something like this: https://github.com/python/cpython/blob/3.10/Lib/ftplib.py

3. Create a copy of this code into your project (example: my_lib\my_ftplib.py)

4. For the method that is failing, in your case STORBINARY, the error looks to be on the line where it says conn.unwrap() in that method. Comment this line. Enter keyword pass otherwise the empty if block will give syntax error.

5. Import the above library in your file where you are instantiating the FTP_TLS. Now you will no longer face this error.

Reasoning: The code in the function def ntransfercmd (under FTP_LTS class) encloses the conn object into a SSL session. The above line which you have commented is responsible for tearing down the SSL session post transfer. For some reason, when using Microsoft's FTP server, the code gets blocked on that line and results in timeout. This can be either because post transfer the server drops the connection or maybe the server unwraps the SSL from its side. I am not sure. Commenting that line is harmless because eventually the connection will be closed anyways - see below for details:

In ftplib's python code, you will notice that the conn object in STORBINARY function is enclosed in a with block, and that it is created using socket.create_connection. This means that .close() is automatically called when the code exits the with block (you can confirm this by looking at the __exit__ method on source code of python's socket class).