IIS logs show sc-win32-status=64 but only through some networks

I have an ASP.NET application running on a client server (W2k3, IIS6, .NET 2.0). FWIW, this is a Test instance, it hasn't been moved into Production yet. So it is not running under SSL, load balancing, etc.

When I access one of the pages on their server from our office, the page gets hit once. Inspecting the IIS logs (c:WINDOWS\system32\LogFiles\W3SVC1) show a GET for that page, then I push a button on the page and the log file shows a POST. This seems to be working fine so far.

Now when I remote into the client's network and access the page from one of their local machines, the log file shows a GET, then I push the button on the page and the log shows two POSTs at the same second. The first one shows status (sc-status, sc-substatus, sc-win32-status) 200 0 64, the second shows 200 0 0.

In the log file, both POSTs are identical. Basically the log looks like this (except I masked some of the data):

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status 
2009-08-11 20:19:32 x.x.x.x GET /File.aspx - 80 - y.y.y.y Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.0;+WOW64;+Trident/4.0;+SLCC1;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.21022;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30618;+MDDR;+OfficeLiveConnector.1.4;+OfficeLivePatch.0.0) 200 0 0
2009-08-11 20:19:45 x.x.x.x POST /File.aspx - 80 - y.y.y.y Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.0;+WOW64;+Trident/4.0;+SLCC1;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.21022;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30618;+MDDR;+OfficeLiveConnector.1.4;+OfficeLivePatch.0.0) 200 0 64
2009-08-11 20:19:45 x.x.x.x POST /File.aspx - 80 - y.y.y.y Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.0;+WOW64;+Trident/4.0;+SLCC1;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.21022;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30618;+MDDR;+OfficeLiveConnector.1.4;+OfficeLivePatch.0.0) 200 0 0

The problem is, the page is getting hit twice. The database performs an operation for the first request, then the second request detects that a duplicate operation is being performed and throws an error message. The users think their operation failed, but it actually succeeded.

The error description of sc-win32-status 64 is: "The specified network name is no longer available." This leads me to believe, given that both POST requests show an HTTP status of 200, that the server is successful in serving the request, but the client is never notified and resubmits the request.

• How can I troubleshoot this?

• Any ideas what could be causing this behavior on their internal network only?

• I should mention, this is happening at two separate client sites, but does not happen at six of our other client sites, or in our office, or connecting to any of our eight clients over the web.

• What could be making this reproducible 100% of the time on their local network but 0% of the time anywhere else?

Update: I found a very small number of the duplicated POST requests had sc-win32-status of 995 instead of 64 as originally reported. The error description of sc-win32-status=995 is: "The I/O operation has been aborted because of either a thread exit or an application request." This doesn't make any sense (considering I have full access to the code). I still don't understand how or why this issue is occurring, but the new error code leads me to believe it may not be a network issue after all and I am now investigating the possibility of a random code bug.

Solution 1:

This is my understanding of the issue so far:

• sc-win32-status 64 means “The specified network name is no longer available.”
• After IIS has sent the final response to the client, typically it waits for an ACK message from the client.
• Sometimes clients will reset the connection instead of sending the final ACK back to server. This is not a graceful connection close, so IIS logs the “64” code.
• Many clients will reset the connection when they are done with it, to free up the socket instead of leaving it in TIME_WAIT/CLOSE_WAIT.
• Proxies tend to do this more than others do.

Update: I found some interesting information here and here, so I basically re-wrote the page to ensure there wasn't any bad markup etc. and... the problem is now gone! It was just a shot in the dark, and I couldn't definitely say what it was that fixed the problem, since it was only affecting some of our clients under some very specific circumstances...

Solution 2:

I experienced this same issue when trying to serve up gzipped binary files from IIS6 through a proxy server. I wasn't experiencing any problem when going to the website directly.

I found this was the cause in my case by running Fiddler on a client machine and inspecting the response. Fiddler warns that the response is encoded and then complains that the magic number on the gzip file wasn't correct.

I turned off gzip compression for binary files in my code and the issue stopped occurring.