Cloudflare and AWS - Intermittent 525 SSL Handshake

Solution tldr;

We use ESET File Security 7 on our servers and it was creating a temporary blacklist of IPs, which sometimes included Cloudflare ones. I added Cloudflare IPs to the IDS exception.

Detail

The Cloudflare support engineer (Andronicus - thank you!) found the problem to be intermittent from within Cloudflare and suggested looking for anything that would do a dynamic blocking of IPs - especially Cloudflare's. As all our traffic comes through Cloudflare, it might look like an attack and attackers hitting the domain endpoint would also have a Cloudflare IP.

I took that and did a lot more reading. We're on AWS EC2, so the technology they use is AWS Shield, which is on by default and deals with flood rather than our low levels.

I then did a full inventory of the server. Went through every single app and Windows Firewall in detail. I have plenty of experience with Windows Firewall, so I could see that there was nothing misconfigured.

ESET File Security for Windows Server is a pretty good anti-virus - particularly for botnet intrusion detection. It also has a feature called Network Attack Protection (IDS) for scanning for suspicious network traffic. Part of that is a "temporary IP address blacklist". From ESET:

View a list of IP addresses that have been detected as the source of attacks and added to the blacklist to block connections for a certain period of time. Shows IP address that have been locked.

IPs get added to the blacklist for a short period of time. By watching the blacklist (it doesn't keep logs) I spotted some familiar Cloudflare IPs popping up.

I've added the Cloudflare IPs to the IDS exception list (it allows address ranges) and raised a ticket with ESET to ask them how I keep that list up to date as Cloudflare can't be expected to keep the full list of IPs static.

If that doesn't help you, check out my full post on Cloudflare Community.