How to connect to FTPS server with data connection using same TLS session?

Environment: I'm using Sun Java JDK 1.8.0_60 on 64-bit Windows 7, using Spring Integration 4.1.6 (which internally appears to use Apache Commons Net 3.3 for FTPS access).

I'm attempting to integrate with our application an automatic download from our client's FTPS server. I've done so successfully with SFTP servers using Spring Integration without any trouble for other clients without issues, but this is the first time a client has required us to use FTPS, and getting it to connect has been very puzzling. While in my real application I'm configuring Spring Integration using XML beans, to try to understand what's not working I'm using the following test code (though I'm anonymizing the actual host/username/password here):

final DefaultFtpsSessionFactory sessionFactory = new DefaultFtpsSessionFactory();
sessionFactory.setHost("XXXXXXXXX");
sessionFactory.setPort(990);
sessionFactory.setUsername("XXXXXXX");
sessionFactory.setPassword("XXXXXXX");
sessionFactory.setClientMode(2);
sessionFactory.setFileType(2);
sessionFactory.setUseClientMode(true);
sessionFactory.setImplicit(true);
sessionFactory.setTrustManager(TrustManagerUtils.getAcceptAllTrustManager());
sessionFactory.setProt("P");
sessionFactory.setProtocol("TLSv1.2");
sessionFactory.setProtocols(new String[]{"TLSv1.2"});
sessionFactory.setSessionCreation(true);
sessionFactory.setCipherSuites(new String[]{"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"});

final FtpSession session = sessionFactory.getSession();
//try {
    final FTPFile[] ftpFiles = session.list("/");
    logger.debug("FtpFiles: {}", (Object[]) ftpFiles);
//} catch (Exception ignored ) {}
session.close();

I'm running this code with -Djavax.net.debug=all to get all the TLS debugging information printed.

The main "control" connection to the FTPS server works fine, but when it tries to open the data connection for the list (or any other data connection I've tried), I get a javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake, caused by java.io.EOFException: SSL peer shut down incorrectly. If I uncomment the swallowing-exceptions catch block around the session.list command, then I can see (though the javax.net.debug output) that the server sent the following message after rejecting the data connection SSL handshake:

main, READ: TLSv1.2 Application Data, length = 129
Padded plaintext after DECRYPTION:  len = 105
0000: 34 35 30 20 54 4C 53 20   73 65 73 73 69 6F 6E 20  450 TLS session 
0010: 6F 66 20 64 61 74 61 20   63 6F 6E 6E 65 63 74 69  of data connecti
0020: 6F 6E 20 68 61 73 20 6E   6F 74 20 72 65 73 75 6D  on has not resum
0030: 65 64 20 6F 72 20 74 68   65 20 73 65 73 73 69 6F  ed or the sessio
0040: 6E 20 64 6F 65 73 20 6E   6F 74 20 6D 61 74 63 68  n does not match
0050: 20 74 68 65 20 63 6F 6E   74 72 6F 6C 20 63 6F 6E   the control con
0060: 6E 65 63 74 69 6F 6E 0D   0A                       nection..

What appears to be happening (and this is my first time dealing with FTPS, though I've dealt with plain FTP before) is that the way the server ensures authentication and encryption over both the control and data connections is that after a "normal" TLS connection to establish the control connection and authentication happens there, each data connection requires the client to connect with the same TLS session. This makes sense to me as how it's supposed to be work, but the Apache Commons Net FTPS implementation doesn't seem to be doing that. It seems to be trying to establish a new TLS session, and so the server is rejecting the attempt.

Based on this question about resuming SSL sessions in JSSE, it appears that Java assumes or requires a different session for each host/post combination. My hypothesis is that since the FTPS data connection is on a different port than the control connection, it's not finding the existing session and is trying to establish a new one, so the connection fails.

I see three main possibilities:

The server is not following the FTPS standard in requiring the same TLS session on the data port as on the control port. I can connect to the server fine (using the same host/user/password as I'm trying to use in my code) using FileZilla 3.13.1. The server identifies itself as "FileZilla Server 0.9.53 beta" upon login, so perhaps this is some sort of proprietary FileZilla way of doing things, and there's something odd I need to do to convince Java to use the same TLS session.
The Apache Commons Net client doesn't actually follow the FTPS standard, and only allows some subset that doesn't allow for securing the data connections. This would seem odd, as it appears to be the standard way of connecting to FTPS from within Java.
I'm completely missing something and misdiagnosing this.

I'd appreciate any direction you can provide as to how to connect to this kind of FTPS server. Thank you.

Indeed some FTP(S) servers do require that the TLS/SSL session is reused for the data connection. This is a security measure by which the server can verify that the data connection is used by the same client as the control connection.

Some references for common FTP servers:

vsftpd: https://scarybeastsecurity.blogspot.com/2009/02/vsftpd-210-released.html
FileZilla server: https://svn.filezilla-project.org/filezilla?view=revision&revision=6661
ProFTPD: http://www.proftpd.org/docs/contrib/mod_tls.html#TLSOptions (NoSessionReuseRequired directive)

What may help you with the implementation is that Cyberduck FTP(S) client does support TLS/SSL session reuse and it uses Apache Commons Net library:

https://github.com/iterate-ch/cyberduck/issues/5087 – Reuse Session key on data connection

See its FTPClient.java code (extends Commons Net FTPSClient), particularly its override of _prepareDataSocket_ method:

 @Override
 protected void _prepareDataSocket_(final Socket socket) {
     if(preferences.getBoolean("ftp.tls.session.requirereuse")) {
         if(socket instanceof SSLSocket) {
             // Control socket is SSL
             final SSLSession session = ((SSLSocket) _socket_).getSession();
             if(session.isValid()) {
                 final SSLSessionContext context = session.getSessionContext();
                 context.setSessionCacheSize(preferences.getInteger("ftp.ssl.session.cache.size"));
                 try {
                     final Field sessionHostPortCache = context.getClass().getDeclaredField("sessionHostPortCache");
                     sessionHostPortCache.setAccessible(true);
                     final Object cache = sessionHostPortCache.get(context);
                     final Method putMethod = cache.getClass().getDeclaredMethod("put", Object.class, Object.class);
                     putMethod.setAccessible(true);
                     Method getHostMethod;
                     try {
                         getHostMethod = socket.getClass().getMethod("getPeerHost");
                     }
                     catch(NoSuchMethodException e) {
                         // Running in IKVM
                         getHostMethod = socket.getClass().getDeclaredMethod("getHost");
                     }
                     getHostMethod.setAccessible(true);
                     Object peerHost = getHostMethod.invoke(socket);
                     putMethod.invoke(cache, String.format("%s:%s", peerHost, socket.getPort()).toLowerCase(Locale.ROOT), session);
                 }
                 catch(NoSuchFieldException e) {
                     // Not running in expected JRE
                     log.warn("No field sessionHostPortCache in SSLSessionContext", e);
                 }
                 catch(Exception e) {
                     // Not running in expected JRE
                     log.warn(e.getMessage());
                 }
             }
             else {
                 log.warn(String.format("SSL session %s for socket %s is not rejoinable", session, socket));
             }
         }
     }
 }

It seems that the _prepareDataSocket_ method was added to Commons Net FTPSClient specifically to allow the TLS/SSL session reuse implementation:
https://issues.apache.org/jira/browse/NET-426

A native support for the reuse is still pending:
https://issues.apache.org/jira/browse/NET-408
You will obviously need to override the Spring Integration DefaultFtpsSessionFactory.createClientInstance() to return your custom FTPSClient implementation with the session reuse support.

The above solution does not work on its own anymore since JDK 8u161.

According to JDK 8u161 Update Release Notes (and the answer by @Laurent):

Added TLS session hash and extended master secret extension support
...
In case of compatibility issues, an application may disable negotiation of this extension by setting the System Property jdk.tls.useExtendedMasterSecret to false in the JDK

I.e., you can call this to fix the problem (you still need to override the _prepareDataSocket_):

System.setProperty("jdk.tls.useExtendedMasterSecret", "false");

Though this should be a considered a workaround only. I do not know a proper solution.

An alternative implementation is here:
https://issues.apache.org/jira/browse/NET-408

There's a separate question about problems in 1.8.0_161:
SSL Session reuse in Apache FTPS client in JDK 8u161

I actually had the same problem in the past (just in C++/OpenSSL, I do not do Java), so I knew what to google for.

You can use this SSLSessionReuseFTPSClient class :

import java.io.IOException;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.net.Socket;
import java.util.Locale;

import javax.net.ssl.SSLSession;
import javax.net.ssl.SSLSessionContext;
import javax.net.ssl.SSLSocket;

import org.apache.commons.net.ftp.FTPSClient;

public class SSLSessionReuseFTPSClient extends FTPSClient {

    // adapted from:
    // https://trac.cyberduck.io/browser/trunk/ftp/src/main/java/ch/cyberduck/core/ftp/FTPClient.java
    @Override
    protected void _prepareDataSocket_(final Socket socket) throws IOException {
        if (socket instanceof SSLSocket) {
            // Control socket is SSL
            final SSLSession session = ((SSLSocket) _socket_).getSession();
            if (session.isValid()) {
                final SSLSessionContext context = session.getSessionContext();
                try {
                    final Field sessionHostPortCache = context.getClass().getDeclaredField("sessionHostPortCache");
                    sessionHostPortCache.setAccessible(true);
                    final Object cache = sessionHostPortCache.get(context);
                    final Method method = cache.getClass().getDeclaredMethod("put", Object.class, Object.class);
                    method.setAccessible(true);
                    method.invoke(cache, String
                            .format("%s:%s", socket.getInetAddress().getHostName(), String.valueOf(socket.getPort()))
                            .toLowerCase(Locale.ROOT), session);
                    method.invoke(cache, String
                            .format("%s:%s", socket.getInetAddress().getHostAddress(), String.valueOf(socket.getPort()))
                            .toLowerCase(Locale.ROOT), session);
                } catch (NoSuchFieldException e) {
                    throw new IOException(e);
                } catch (Exception e) {
                    throw new IOException(e);
                }
            } else {
                throw new IOException("Invalid SSL Session");
            }
        }
    }
}

And With openJDK 1.8.0_161 :

We must set :

System.setProperty("jdk.tls.useExtendedMasterSecret", "false");

according to http://www.oracle.com/technetwork/java/javase/8u161-relnotes-4021379.html

Added TLS session hash and extended master secret extension support

In case of compatibility issues, an application may disable negotiation of this extension by setting the System Property jdk.tls.useExtendedMasterSecret to false in the JDK

How to connect to FTPS server with data connection using same TLS session?

Related

Recent Posts