Why would an SMTP client send a RSET (Reset) command?

Solution 1:

TLDR

I had been wondering why an SMTP client would interrupt its own transaction by issuing RSET commands instead of the DATA command. The short answer is that it wouldn’t; this is a symptom of SMTP connections being intercepted by anti-virus software.

Client Logging

I enabled the Debug option in the configuration for sSMTP but it took some time for me to figure out how to install and configure syslog in Cygwin so that messages from Cygwin processes are logged to /var/log/messages instead of the Windows event viewer.

However, it only logged the first MAIL FROM: and RCPT TO: commands; there was no indication that these commands were sent more than once – or that sSMTP ever sent an RSET command.

As mentioned in my question, I had checked the sSMTP source code but there's no code to send an RSET command.

Symantec Interception of SMTP traffic

User masegaloeh suggested that anti-virus software may be modifying the SMTP packets – and he was right: I temporarily disabled Symantec Endpoint Protection on my computer and the SMTP transactions proceeded as normal.

After re-enabling Symantec Endpoint Protection, I monitored TCP connections using the TCPView utility from Windows Sysinternals and I could see that the Symantec ccSvcHst.exe process was proxying all TCP traffic whose destination port is 25.

I telnetted to Port 25 on the mail server (netcat wouldn’t work properly, possibly due to Symantec interception) to send a test mail using by manually entering SMTP commands. At the same time, I had another terminal window open with an SSH connection to the mail host while running sudo tail -F /var/log/maillog to simultaneously monitor what the SMTP server was seeing.

The interception carried out by the Symantec proxy is subtle. From the mail client’s perspective, there’s very little indication that it’s not talking directly to the SMTP server. Most of the commands are passed through to the mail server as they were sent and the responses are what you’d expect. It wasn’t until I entered the DATA command, that the Symantec proxy started to change things: it responded with:

354 Please start mail input.

This seems normal but in reality, the response from my Postfix server should have been

354 End data with <CR><LF>.<CR><LF>

Also: it didn’t actually pass the DATA command on to Postfix until after I had completed the message body and followed it with a QUIT.

Note: while I was typing the contents of the test message, the Symantec proxy kept its connection to the Postfix server alive by issuing NOOP commands.

Dealing with buggy SMTP clients

I mentioned in my question that my eventual goal was to troubleshoot a proprietary (binary, closed source) mail client used by my organisation. I discovered that this client is really buggy: it sends an invalid HELO command (without any hostname) and then simply gives up and QUITs after it’s politely informed by the SMTP server that its syntax is wrong – even though I had configured Postfix to not require a HELO (valid or otherwise).

I resolved this issue by installing a new server with CentOS 7 which comes with a Postfix version recent enough so that I can disable postfix HELO checks completely (similar to how MS Exchange simply ignores invalid HELO commands).

General use of RSET

I had also been wondering about the general usage of RSET in SMTP transactions and I found the following in the original RFC 821 for SMTP:

This command specifies that the current mail transaction is to be aborted. Any stored sender, recipients, and mail data must be discarded, and all buffers and state tables cleared. The receiver must send an OK reply.

RSET would be used if it turned out that the recipient email address was for a non-existent user. The following SMTP transaction is an example of an Aborted SMTP Transaction Scenario.

R: 220 MIT-Multics.ARPA Simple Mail Transfer Service Ready
S: HELO ISI-VAXA.ARPA
R: 250 MIT-Multics.ARPA

S: MAIL FROM:<[email protected]>
R: 250 OK

S: RCPT TO:<[email protected]>
R: 250 OK

S: RCPT TO:<[email protected]>
R: 550 No such user here

S: RSET
R: 250 OK

S: QUIT
R: 221 MIT-Multics.ARPA Service closing transmission channel

Re-use of SMTP connections

SMTP clients can use the same SMTP connection to send multiple messages to the same destination. This feature is termed SMTP connection caching by Postfix. When using this performance feature, Postfix sends an RSET before each MAIL FROM command to verify that the SMTP connection is still usable (see Postfix Connection Cache).