Windows Filtering Platform dropping SQL Server connections
I've been investigating connection issues between my web server (Web01) and a database server (Database01). My current setup:
Web01 - two NICs, one external (firewalled), one internal (not firewalled).
Database01 - Same configuration as above.
The two servers communicate on using the private NIC, so the firewall profile is disabled. What I am finding, is that I get two errors at Web01:
WEB01: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)
or
WEB01: The wait operation timed out
This is not consistent, although they are more likely to occur when I am running a crawl (via Xenu) on Web01 to crawl through a new website.
On Database01, I have identified a series of messages from the Windows Filtering Platform:
DATABASE01: The Windows Filtering Platform has blocked a packet.
Researching into this, its the silent Port Scanning Prevention Filter built into the Windows Firewall. This feature runs regardless of the fact that the Private profile (for the private NIC) is turned off.
The intention of this feature, is to intercept requests for ports where no listener is attached, and reject the packet.
The problem with this, is the port in question is the default sql server port, 1433 and sql server is listening.
So my question would be, under what other circumstances would the firewall drop packets this way.
Other comments/possible problems/notes:
- I am using Entity Framework, with Autofac and my context uses a per-request lifetime scope. This should ensure connections are closed and disposed of at the end of each request, but internally this implements a Unit-of-Work model, whereby I hope connections are opened and closed for each batch operation, therefore shouldn't wait for the end of any potentially long running requests.
- The database pool configuration for ASP.NET hasn't been changed, so connection lifetimes and connection pool size is all set to default.
- The connection string originally just specified the server name, therefore the default instance:
server=database01, but to eliminate any potential issues with the Sql Server Browser service, I've modified the connection string toserver=tcp:database01,1433to enforce the TCP connection is used (so it doesn't bother trying shared memory or named pipes), and have specified the port exactly (so it doesn't need to query the browser service first to identity the right port). - Database01 does act as a Principal for database mirroring (as part of a 2-server mirror solution with automatic failover, Database02 as the partner, and Web01 running SQL Server Express as the witness). I've disabled mirroring of the target databases and the problem still occurs.
- I've confirmed that
sqlserver.exeis listening on port1433using thenetstat -anob -p tcpcommand and can see it bound as follows:
netstat -anob -p tcp
TCP 0.0.0.0:1433 0.0.0.0:0 LISTENING 1896
TCP 192.168.3.2:1433 192.168.3.2:49274 ESTABLISHED 1896
In the above results, 192.168.3.2 is the private IP for Database01 and 192.168.3.1 is the private IP for Web01, with external port 49274 being dynamically assigned by the ASP.NET connection pool.
I've tried to eliminate as much as possible, but I still can't determine the root problem:
Why is the Windows Firewall dropping packets for a port that is being listened on?
Web01: Windows Server 2012 R2 Standard 64bit
Database01: Windows Server 2012 64bit SQL Server 2012 SP1 Standard
The solution to this issue was a feature called TCP Offloading. I'm not sure if this is because I am running in a virtualised environment or not, but TCP Offloading was causing the packet drops. When this feature is disabled the drop packets do seem to disappear.
TCP Offloading removes the burden of processing network I/O on the CPU by offloading it onto the NIC.
I do wonder if I was running a hardware environment if this issue would still occur.
https://support.rackspace.com/how-to/disabling-tcp-offloading-in-windows-server-2012/