Cannot bind to some ports due to permission denied
For the last 3 months or so I'm having random errors where I can't bind a specific port where our Identity server is running on my local development workstation. At first I thought it's my broken machine, so I reset everything, which kinda fixed the issue for 2 months and now it is back.
In the meanwhile other developers saw the same issue. All of us who experience the issue are running Windows 10, but not everyone with Windows 10 has that issue.
Restarting windows after it was shut down with shutdown /s /f /t 0 is the only reliable solution to open up the port again. After I restart normally (due to windows updates or similar) the issue comes up again an I need to shut down windows with that command.
I wrote a small f# script to test which ports are affected. The exact error message I get is
An attempt was made to access a socket in a way forbidden by its access permissions
When I looked for it, the suggested solutions where to restart the machine, but that seems more like a work around, not a real solution. I've also found https://stackoverflow.com/a/10461978/621366 - but netstat -o doesn't list the port, neither does the tool TCPView. All of us are pretty sure that the port isn't occupied by anything. I also tried running netstat in an administrator console and similar commands inside bash on windows, but couldn't find anything. I can't even connect to the port via telnet, it says nothing is listening on the port.
those ports in question are for instance:
- 49670 - 49689
- 49710 - 49749
- 49760 - 49779
- 49811 - 49830
- 49843 - 49882
- 50197 - 50216
None of us modified anything on the windows firewall or has any additional anti virus tools installed except the windows 10 default ones. So everything should be on default values. And it also worked normally for ~10 months before it broke the first time and afterwards for 2 months. In both cases after some windows updates where installed. The last time it was a bios update (probably due to the meltdown / spectre issues?).
Also trying to open up the ports explicitly on the firewall didn't help.
According to this answer https://stackoverflow.com/a/23982901/621366 TCPView and netstat shouldn't miss any occupied ports, but even when I enable showing unconnected endpoints in TCPView, I don't see any of the ports where I get permission denied when trying to bind them. Here a screen from the occupied ports (I marked the bordering occupied ports which are right before or after the group of permission-denied-ports)
UPDATE:
I've noticed that it always seems to be 160 or 180 (exact numbers) of ports which have permission denied in the ranges of 40,000+ This seems oddly coincidental to me, so obviously something is occupying the ports on purpose, but what? I can't seem to find anything in the windows event logs (although I wouldn't know what to look for exactly) and none of those ports shows up any any of my firewall rules. Also shutting down docker for windows doesn't make any difference and when a colleague mentioned that for them it's enough to restart docker for windows (in the UI go to Reset->Restart) and right now for me, even restarting with the shutdown command doesn't work anymore.
UPDATE 2:
The output of netstat -ano run from an administrator powershell:
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 1152
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:2179 0.0.0.0:0 LISTENING 4696
TCP 0.0.0.0:5040 0.0.0.0:0 LISTENING 6616
TCP 0.0.0.0:5357 0.0.0.0:0 LISTENING 4
TCP 0.0.0.0:5432 0.0.0.0:0 LISTENING 11100
TCP 0.0.0.0:7680 0.0.0.0:0 LISTENING 7056
TCP 0.0.0.0:17500 0.0.0.0:0 LISTENING 9668
TCP 0.0.0.0:49664 0.0.0.0:0 LISTENING 784
TCP 0.0.0.0:49665 0.0.0.0:0 LISTENING 1628
TCP 0.0.0.0:49666 0.0.0.0:0 LISTENING 2028
TCP 0.0.0.0:49667 0.0.0.0:0 LISTENING 3560
TCP 0.0.0.0:49800 0.0.0.0:0 LISTENING 856
TCP 0.0.0.0:49821 0.0.0.0:0 LISTENING 892
TCP 0.0.0.0:50000 0.0.0.0:0 LISTENING 11100
TCP 0.0.0.0:50001 0.0.0.0:0 LISTENING 11100
TCP 0.0.0.0:51000 0.0.0.0:0 LISTENING 11100
TCP 10.0.75.1:139 0.0.0.0:0 LISTENING 4
TCP 10.0.75.1:445 10.0.75.2:44848 ESTABLISHED 4
TCP 127.0.0.1:843 0.0.0.0:0 LISTENING 9668
TCP 127.0.0.1:944 0.0.0.0:0 LISTENING 688
TCP 127.0.0.1:944 127.0.0.1:50968 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50970 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50973 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50977 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50981 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50990 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50992 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:50996 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51005 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51007 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51009 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51015 TIME_WAIT 0
TCP 127.0.0.1:944 127.0.0.1:51017 ESTABLISHED 688
TCP 127.0.0.1:4380 0.0.0.0:0 LISTENING 11024
TCP 127.0.0.1:6942 0.0.0.0:0 LISTENING 9296
TCP 127.0.0.1:17600 0.0.0.0:0 LISTENING 9668
TCP 127.0.0.1:49668 127.0.0.1:49669 ESTABLISHED 688
TCP 127.0.0.1:49669 127.0.0.1:49668 ESTABLISHED 688
TCP 127.0.0.1:50076 127.0.0.1:50077 ESTABLISHED 8828
TCP 127.0.0.1:50077 127.0.0.1:50076 ESTABLISHED 8828
TCP 127.0.0.1:50173 127.0.0.1:50174 ESTABLISHED 9668
TCP 127.0.0.1:50174 127.0.0.1:50173 ESTABLISHED 9668
TCP 127.0.0.1:50175 127.0.0.1:50176 ESTABLISHED 9668
TCP 127.0.0.1:50176 127.0.0.1:50175 ESTABLISHED 9668
TCP 127.0.0.1:50197 127.0.0.1:50198 ESTABLISHED 9668
TCP 127.0.0.1:50198 127.0.0.1:50197 ESTABLISHED 9668
TCP 127.0.0.1:50335 127.0.0.1:50336 ESTABLISHED 6424
TCP 127.0.0.1:50336 127.0.0.1:50335 ESTABLISHED 6424
TCP 127.0.0.1:50346 127.0.0.1:50347 ESTABLISHED 11100
TCP 127.0.0.1:50347 127.0.0.1:50346 ESTABLISHED 11100
TCP 127.0.0.1:51011 127.0.0.1:51012 ESTABLISHED 9296
TCP 127.0.0.1:51012 127.0.0.1:51011 ESTABLISHED 9296
TCP 127.0.0.1:51013 127.0.0.1:51014 ESTABLISHED 9296
TCP 127.0.0.1:51014 127.0.0.1:51013 ESTABLISHED 9296
TCP 127.0.0.1:51016 0.0.0.0:0 LISTENING 9296
TCP 127.0.0.1:51017 127.0.0.1:944 ESTABLISHED 8828
TCP 127.0.0.1:63342 0.0.0.0:0 LISTENING 9296
TCP 127.94.0.1:946 0.0.0.0:0 LISTENING 688
TCP 127.94.0.2:946 0.0.0.0:0 LISTENING 688
TCP 127.94.0.3:946 0.0.0.0:0 LISTENING 688
TCP 127.94.0.4:946 0.0.0.0:0 LISTENING 688
TCP 169.254.105.83:139 0.0.0.0:0 LISTENING 4
TCP 192.168.0.107:139 0.0.0.0:0 LISTENING 4
TCP 192.168.0.107:49415 111.221.29.134:443 ESTABLISHED 4316
TCP 192.168.0.107:49417 111.221.29.127:443 ESTABLISHED 4316
TCP 192.168.0.107:50185 162.125.66.3:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50246 52.70.31.26:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50253 35.177.204.73:443 ESTABLISHED 2804
TCP 192.168.0.107:50254 35.177.204.73:443 ESTABLISHED 2804
TCP 192.168.0.107:50256 35.177.204.73:443 ESTABLISHED 2804
TCP 192.168.0.107:50257 158.85.224.175:443 ESTABLISHED 10836
TCP 192.168.0.107:50258 13.69.14.160:443 ESTABLISHED 8620
TCP 192.168.0.107:50310 66.102.1.188:443 ESTABLISHED 11184
TCP 192.168.0.107:50329 157.240.20.15:443 ESTABLISHED 10836
TCP 192.168.0.107:50331 111.221.29.74:443 ESTABLISHED 10072
TCP 192.168.0.107:50332 162.125.18.133:443 ESTABLISHED 9668
TCP 192.168.0.107:50351 40.77.226.194:443 ESTABLISHED 8620
TCP 192.168.0.107:50460 66.102.1.189:443 ESTABLISHED 10836
TCP 192.168.0.107:50470 66.102.1.189:443 ESTABLISHED 10836
TCP 192.168.0.107:50501 192.30.253.125:443 ESTABLISHED 11184
TCP 192.168.0.107:50513 40.77.226.194:443 ESTABLISHED 8620
TCP 192.168.0.107:50529 87.98.218.198:443 ESTABLISHED 12540
TCP 192.168.0.107:50530 172.217.21.46:443 ESTABLISHED 10836
TCP 192.168.0.107:50616 172.217.21.46:443 ESTABLISHED 10836
TCP 192.168.0.107:50630 162.125.18.133:443 ESTABLISHED 9668
TCP 192.168.0.107:50641 172.217.21.37:443 ESTABLISHED 10836
TCP 192.168.0.107:50645 162.125.66.4:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50668 87.98.218.198:443 ESTABLISHED 12540
TCP 192.168.0.107:50703 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50709 192.30.253.125:443 ESTABLISHED 11184
TCP 192.168.0.107:50744 87.98.218.198:443 ESTABLISHED 12540
TCP 192.168.0.107:50828 162.125.66.3:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50830 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50831 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50832 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50834 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50835 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50836 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50837 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50839 192.30.253.125:443 ESTABLISHED 11184
TCP 192.168.0.107:50844 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50847 192.30.253.124:443 ESTABLISHED 11184
TCP 192.168.0.107:50857 192.30.253.124:443 ESTABLISHED 11184
TCP 192.168.0.107:50863 162.125.34.137:443 CLOSE_WAIT 9668
TCP 192.168.0.107:50865 172.217.21.46:443 TIME_WAIT 0
TCP 192.168.0.107:50866 172.217.21.46:443 ESTABLISHED 10836
TCP 192.168.0.107:50910 35.186.213.138:443 TIME_WAIT 0
TCP 192.168.0.107:50923 172.217.21.46:443 ESTABLISHED 10836
TCP 192.168.0.107:50925 40.117.190.72:443 ESTABLISHED 4040
TCP 192.168.0.107:50927 172.217.21.42:443 ESTABLISHED 11184
TCP 192.168.0.107:50949 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50950 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50951 151.101.0.133:443 ESTABLISHED 11184
TCP 192.168.0.107:50954 192.30.253.124:443 ESTABLISHED 11184
TCP 192.168.0.107:50959 40.117.190.72:443 TIME_WAIT 0
TCP 192.168.0.107:50969 192.30.253.113:22 TIME_WAIT 0
TCP 192.168.0.107:50978 87.98.218.198:443 ESTABLISHED 12540
TCP 192.168.0.107:50984 151.101.1.69:443 ESTABLISHED 11184
TCP 192.168.0.107:50985 192.0.73.2:443 ESTABLISHED 11184
TCP 192.168.0.107:50986 104.16.112.18:443 ESTABLISHED 11184
TCP 192.168.0.107:50991 198.252.206.25:443 ESTABLISHED 11184
TCP 192.168.0.107:50993 192.168.0.10:3910 TIME_WAIT 0
TCP 192.168.0.107:50994 192.168.0.10:3910 TIME_WAIT 0
TCP 192.168.0.107:50997 23.210.254.37:443 ESTABLISHED 912
TCP 192.168.0.107:50998 23.210.254.37:443 ESTABLISHED 912
TCP 192.168.0.107:50999 23.210.254.37:443 ESTABLISHED 912
TCP 192.168.0.107:51001 23.210.254.37:443 ESTABLISHED 912
TCP 192.168.0.107:51006 40.117.190.72:443 ESTABLISHED 11992
TCP 192.168.0.107:51008 40.69.218.62:443 ESTABLISHED 7056
TCP 192.168.0.107:51010 172.217.21.46:443 ESTABLISHED 11184
TCP [::]:135 [::]:0 LISTENING 1152
TCP [::]:445 [::]:0 LISTENING 4
TCP [::]:2179 [::]:0 LISTENING 4696
TCP [::]:5357 [::]:0 LISTENING 4
TCP [::]:7680 [::]:0 LISTENING 7056
TCP [::]:17500 [::]:0 LISTENING 9668
TCP [::]:49664 [::]:0 LISTENING 784
TCP [::]:49665 [::]:0 LISTENING 1628
TCP [::]:49666 [::]:0 LISTENING 2028
TCP [::]:49667 [::]:0 LISTENING 3560
TCP [::]:49800 [::]:0 LISTENING 856
TCP [::]:49821 [::]:0 LISTENING 892
TCP [::1]:5432 [::]:0 LISTENING 11100
TCP [::1]:50000 [::]:0 LISTENING 11100
TCP [::1]:50001 [::]:0 LISTENING 11100
TCP [::1]:51000 [::]:0 LISTENING 11100
UDP 0.0.0.0:53 *:* 5620
UDP 0.0.0.0:3702 *:* 2084
UDP 0.0.0.0:3702 *:* 2084
UDP 0.0.0.0:5050 *:* 6616
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 3080
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5353 *:* 11184
UDP 0.0.0.0:5355 *:* 3080
UDP 0.0.0.0:17500 *:* 9668
UDP 0.0.0.0:49670 *:* 2084
UDP 0.0.0.0:57329 *:* 5620
UDP 0.0.0.0:57330 *:* 5620
UDP 0.0.0.0:59529 *:* 8620
UDP 0.0.0.0:60605 *:* 11184
UDP 10.0.75.1:137 *:* 4
UDP 10.0.75.1:138 *:* 4
UDP 10.0.75.1:1900 *:* 2620
UDP 10.0.75.1:61326 *:* 2620
UDP 127.0.0.1:1900 *:* 2620
UDP 127.0.0.1:60816 *:* 4616
UDP 127.0.0.1:61328 *:* 2620
UDP 169.254.105.83:137 *:* 4
UDP 169.254.105.83:138 *:* 4
UDP 169.254.105.83:1900 *:* 2620
UDP 169.254.105.83:61330 *:* 2620
UDP 172.30.146.241:67 *:* 5620
UDP 172.30.146.241:68 *:* 5620
UDP 172.30.146.241:1900 *:* 2620
UDP 172.30.146.241:61329 *:* 2620
UDP 192.168.0.107:137 *:* 4
UDP 192.168.0.107:138 *:* 4
UDP 192.168.0.107:1900 *:* 2620
UDP 192.168.0.107:61327 *:* 2620
UDP [::]:3702 *:* 2084
UDP [::]:3702 *:* 2084
UDP [::]:5353 *:* 11184
UDP [::]:5353 *:* 11184
UDP [::]:5353 *:* 3080
UDP [::]:5353 *:* 11184
UDP [::]:5355 *:* 3080
UDP [::]:49671 *:* 2084
UDP [::]:57331 *:* 5620
UDP [::]:59529 *:* 8620
UDP [::1]:1900 *:* 2620
UDP [::1]:61323 *:* 2620
UDP [fe80::30eb:ad8f:f94a:b774%26]:1900 *:* 2620
UDP [fe80::30eb:ad8f:f94a:b774%26]:61324 *:* 2620
UDP [fe80::718c:22bb:fd97:c06c%23]:1900 *:* 2620
UDP [fe80::718c:22bb:fd97:c06c%23]:61322 *:* 2620
UDP [fe80::85d0:3b5c:7746:6953%5]:1900 *:* 2620
UDP [fe80::85d0:3b5c:7746:6953%5]:61325 *:* 2620
The f# code I used to test for open ports:
open System.Net
open System.Net.Sockets
let ipAddress = IPAddress([| (byte)0; (byte)0; (byte)0; (byte)0 |])
let ipEndpoint portNumber = (IPEndPoint(ipAddress, portNumber), portNumber)
let getPorts = seq { for i in 1 .. 65535 -> i }
let checkIfPortAvailable (endpoint, portNumber) =
use listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)
try
listener.Bind(endpoint)
(portNumber, true, null)
with
| ex -> (portNumber, false, ex)
[<EntryPoint>]
let main argv =
getPorts
|> Seq.map ipEndpoint
|> Seq.map checkIfPortAvailable
|> Seq.where (fun (_, works, _) -> not works)
|> Seq.where (fun (_, _, ex) -> ex.Message.Contains("An attempt was made to access a socket in a way forbidden by its access permissions"))
|> Seq.iteri (fun index (port, _, _) -> printfn "%5d: %d" index port)
0
and the output when executing the application (it was executed right after the netstat command):
0: 445
1: 5357
2: 7680
3: 49670
4: 49671
5: 49672
6: 49673
7: 49674
8: 49675
9: 49676
10: 49677
11: 49678
12: 49679
13: 49680
14: 49681
15: 49682
16: 49683
17: 49684
18: 49685
19: 49686
20: 49687
21: 49688
22: 49689
23: 49710
24: 49711
25: 49712
26: 49713
27: 49714
28: 49715
29: 49716
30: 49717
31: 49718
32: 49719
33: 49720
34: 49721
35: 49722
36: 49723
37: 49724
38: 49725
39: 49726
40: 49727
41: 49728
42: 49729
43: 49730
44: 49731
45: 49732
46: 49733
47: 49734
48: 49735
49: 49736
50: 49737
51: 49738
52: 49739
53: 49740
54: 49741
55: 49742
56: 49743
57: 49744
58: 49745
59: 49746
60: 49747
61: 49748
62: 49749
63: 49750
64: 49751
65: 49752
66: 49753
67: 49754
68: 49755
69: 49756
70: 49757
71: 49758
72: 49759
73: 49770
74: 49771
75: 49772
76: 49773
77: 49774
78: 49775
79: 49776
80: 49777
81: 49778
82: 49779
83: 49780
84: 49781
85: 49782
86: 49783
87: 49784
88: 49785
89: 49786
90: 49787
91: 49788
92: 49789
93: 49790
94: 49791
95: 49792
96: 49793
97: 49794
98: 49795
99: 49796
100: 49797
101: 49798
102: 49799
103: 49825
104: 49826
105: 49827
106: 49828
107: 49829
108: 49830
109: 49831
110: 49832
111: 49833
112: 49834
113: 49835
114: 49836
115: 49837
116: 49838
117: 49839
118: 49840
119: 49841
120: 49842
121: 49843
122: 49844
123: 49845
124: 49846
125: 49847
126: 49848
127: 49849
128: 49850
129: 49851
130: 49852
131: 49853
132: 49854
133: 49855
134: 49856
135: 49857
136: 49858
137: 49859
138: 49860
139: 49861
140: 49862
141: 49863
142: 49864
143: 49866
144: 49867
145: 49868
146: 49869
147: 49870
148: 49871
149: 49872
150: 49873
151: 49874
152: 49875
153: 49876
154: 49877
155: 49878
156: 49879
157: 49880
158: 49881
159: 49882
160: 49883
161: 49884
162: 49885
163: 50274
164: 50275
165: 50276
166: 50277
167: 50278
168: 50279
169: 50280
170: 50281
171: 50282
172: 50283
173: 50284
174: 50285
175: 50286
176: 50287
177: 50288
178: 50289
179: 50290
180: 50291
181: 50292
182: 50293
The port which I'd actually want to have opened is 49786 but the amount of ports reported with permission denied by my small testing algorithm, but not allocated when checking netstat is confusing me
UPDATE 3:
Now that simply running shutdown /s /f /t 0 doesn't work anymore, I've found another workaround which seems to be working fairly reliably in https://social.technet.microsoft.com/Forums/windows/en-US/e0cc1b3e-2c63-451a-abc8-8e1e66a23287/an-attempt-was-made-to-access-a-socket-in-a-way-forbidden-by-its-access-permissions?forum=w7itpronetworking - netsh int ip reset and restarting the computer (normal windows 10 restart is enough) and the port was consistently available on the next start so far (but it was just 3 tries)
UPDATE 4:
Summary of the comments:
- setting SO_REUSEADDR in the small f# script makes things only worse
- trying to close skype, VPN applications, dropbox and similar didn't have any effect (yes I actually made sure that the processes are terminated)
Solution 1:
The reason is that Hyper-V takes over these ports, to prevent it from happening do the following:
-
dism.exe /Online /Disable-Feature:Microsoft-Hyper-V(will have to restart) netsh int ipv4 add excludedportrange protocol=tcp startport=<your port> numberofports=1dism.exe /Online /Enable-Feature:Microsoft-Hyper-V /All
The original solution is here.
If after that you still can't bind to the port do the following:
-
netsh http add iplisten 0.0.0.0for IPv4 -
netsh http add iplisten ::for IPv6
More information is here.
Solution 2:
A faulty Windows upgrade is the reason for the problems with reserved tcp ports. See https://github.com/docker/for-win/issues/3171#issuecomment-554587817
With this the problem should be fixed:
netsh int ipv4 set dynamic tcp start=49152 num=16384
That would explain why a clean install fixes these type of problems.
Solution 3:
I also had the Hyper-v/WSL problem above, but found a simpler solution:
I had stopped docker before doing this, not sure if it's necessary.
Run as admin:
net stop winnat
net start winnat