Cannot bind to some ports due to permission denied

For the last 3 months or so I'm having random errors where I can't bind a specific port where our Identity server is running on my local development workstation. At first I thought it's my broken machine, so I reset everything, which kinda fixed the issue for 2 months and now it is back.

In the meanwhile other developers saw the same issue. All of us who experience the issue are running Windows 10, but not everyone with Windows 10 has that issue.

Restarting windows after it was shut down with shutdown /s /f /t 0 is the only reliable solution to open up the port again. After I restart normally (due to windows updates or similar) the issue comes up again an I need to shut down windows with that command.

I wrote a small f# script to test which ports are affected. The exact error message I get is

An attempt was made to access a socket in a way forbidden by its access permissions

When I looked for it, the suggested solutions where to restart the machine, but that seems more like a work around, not a real solution. I've also found https://stackoverflow.com/a/10461978/621366 - but netstat -o doesn't list the port, neither does the tool TCPView. All of us are pretty sure that the port isn't occupied by anything. I also tried running netstat in an administrator console and similar commands inside bash on windows, but couldn't find anything. I can't even connect to the port via telnet, it says nothing is listening on the port.

those ports in question are for instance:

  • 49670 - 49689
  • 49710 - 49749
  • 49760 - 49779
  • 49811 - 49830
  • 49843 - 49882
  • 50197 - 50216

None of us modified anything on the windows firewall or has any additional anti virus tools installed except the windows 10 default ones. So everything should be on default values. And it also worked normally for ~10 months before it broke the first time and afterwards for 2 months. In both cases after some windows updates where installed. The last time it was a bios update (probably due to the meltdown / spectre issues?).

Also trying to open up the ports explicitly on the firewall didn't help.

According to this answer https://stackoverflow.com/a/23982901/621366 TCPView and netstat shouldn't miss any occupied ports, but even when I enable showing unconnected endpoints in TCPView, I don't see any of the ports where I get permission denied when trying to bind them. Here a screen from the occupied ports (I marked the bordering occupied ports which are right before or after the group of permission-denied-ports)

enter image description here

UPDATE:

I've noticed that it always seems to be 160 or 180 (exact numbers) of ports which have permission denied in the ranges of 40,000+ This seems oddly coincidental to me, so obviously something is occupying the ports on purpose, but what? I can't seem to find anything in the windows event logs (although I wouldn't know what to look for exactly) and none of those ports shows up any any of my firewall rules. Also shutting down docker for windows doesn't make any difference and when a colleague mentioned that for them it's enough to restart docker for windows (in the UI go to Reset->Restart) and right now for me, even restarting with the shutdown command doesn't work anymore.

UPDATE 2:

The output of netstat -ano run from an administrator powershell:

Proto  Local Address          Foreign Address        State           PID
TCP    0.0.0.0:135            0.0.0.0:0              LISTENING       1152
TCP    0.0.0.0:445            0.0.0.0:0              LISTENING       4
TCP    0.0.0.0:2179           0.0.0.0:0              LISTENING       4696
TCP    0.0.0.0:5040           0.0.0.0:0              LISTENING       6616
TCP    0.0.0.0:5357           0.0.0.0:0              LISTENING       4
TCP    0.0.0.0:5432           0.0.0.0:0              LISTENING       11100
TCP    0.0.0.0:7680           0.0.0.0:0              LISTENING       7056
TCP    0.0.0.0:17500          0.0.0.0:0              LISTENING       9668
TCP    0.0.0.0:49664          0.0.0.0:0              LISTENING       784
TCP    0.0.0.0:49665          0.0.0.0:0              LISTENING       1628
TCP    0.0.0.0:49666          0.0.0.0:0              LISTENING       2028
TCP    0.0.0.0:49667          0.0.0.0:0              LISTENING       3560
TCP    0.0.0.0:49800          0.0.0.0:0              LISTENING       856
TCP    0.0.0.0:49821          0.0.0.0:0              LISTENING       892
TCP    0.0.0.0:50000          0.0.0.0:0              LISTENING       11100
TCP    0.0.0.0:50001          0.0.0.0:0              LISTENING       11100
TCP    0.0.0.0:51000          0.0.0.0:0              LISTENING       11100
TCP    10.0.75.1:139          0.0.0.0:0              LISTENING       4
TCP    10.0.75.1:445          10.0.75.2:44848        ESTABLISHED     4
TCP    127.0.0.1:843          0.0.0.0:0              LISTENING       9668
TCP    127.0.0.1:944          0.0.0.0:0              LISTENING       688
TCP    127.0.0.1:944          127.0.0.1:50968        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:50970        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:50973        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:50977        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:50981        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:50990        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:50992        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:50996        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:51005        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:51007        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:51009        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:51015        TIME_WAIT       0
TCP    127.0.0.1:944          127.0.0.1:51017        ESTABLISHED     688
TCP    127.0.0.1:4380         0.0.0.0:0              LISTENING       11024
TCP    127.0.0.1:6942         0.0.0.0:0              LISTENING       9296
TCP    127.0.0.1:17600        0.0.0.0:0              LISTENING       9668
TCP    127.0.0.1:49668        127.0.0.1:49669        ESTABLISHED     688
TCP    127.0.0.1:49669        127.0.0.1:49668        ESTABLISHED     688
TCP    127.0.0.1:50076        127.0.0.1:50077        ESTABLISHED     8828
TCP    127.0.0.1:50077        127.0.0.1:50076        ESTABLISHED     8828
TCP    127.0.0.1:50173        127.0.0.1:50174        ESTABLISHED     9668
TCP    127.0.0.1:50174        127.0.0.1:50173        ESTABLISHED     9668
TCP    127.0.0.1:50175        127.0.0.1:50176        ESTABLISHED     9668
TCP    127.0.0.1:50176        127.0.0.1:50175        ESTABLISHED     9668
TCP    127.0.0.1:50197        127.0.0.1:50198        ESTABLISHED     9668
TCP    127.0.0.1:50198        127.0.0.1:50197        ESTABLISHED     9668
TCP    127.0.0.1:50335        127.0.0.1:50336        ESTABLISHED     6424
TCP    127.0.0.1:50336        127.0.0.1:50335        ESTABLISHED     6424
TCP    127.0.0.1:50346        127.0.0.1:50347        ESTABLISHED     11100
TCP    127.0.0.1:50347        127.0.0.1:50346        ESTABLISHED     11100
TCP    127.0.0.1:51011        127.0.0.1:51012        ESTABLISHED     9296
TCP    127.0.0.1:51012        127.0.0.1:51011        ESTABLISHED     9296
TCP    127.0.0.1:51013        127.0.0.1:51014        ESTABLISHED     9296
TCP    127.0.0.1:51014        127.0.0.1:51013        ESTABLISHED     9296
TCP    127.0.0.1:51016        0.0.0.0:0              LISTENING       9296
TCP    127.0.0.1:51017        127.0.0.1:944          ESTABLISHED     8828
TCP    127.0.0.1:63342        0.0.0.0:0              LISTENING       9296
TCP    127.94.0.1:946         0.0.0.0:0              LISTENING       688
TCP    127.94.0.2:946         0.0.0.0:0              LISTENING       688
TCP    127.94.0.3:946         0.0.0.0:0              LISTENING       688
TCP    127.94.0.4:946         0.0.0.0:0              LISTENING       688
TCP    169.254.105.83:139     0.0.0.0:0              LISTENING       4
TCP    192.168.0.107:139      0.0.0.0:0              LISTENING       4
TCP    192.168.0.107:49415    111.221.29.134:443     ESTABLISHED     4316
TCP    192.168.0.107:49417    111.221.29.127:443     ESTABLISHED     4316
TCP    192.168.0.107:50185    162.125.66.3:443       CLOSE_WAIT      9668
TCP    192.168.0.107:50246    52.70.31.26:443        CLOSE_WAIT      9668
TCP    192.168.0.107:50253    35.177.204.73:443      ESTABLISHED     2804
TCP    192.168.0.107:50254    35.177.204.73:443      ESTABLISHED     2804
TCP    192.168.0.107:50256    35.177.204.73:443      ESTABLISHED     2804
TCP    192.168.0.107:50257    158.85.224.175:443     ESTABLISHED     10836
TCP    192.168.0.107:50258    13.69.14.160:443       ESTABLISHED     8620
TCP    192.168.0.107:50310    66.102.1.188:443       ESTABLISHED     11184
TCP    192.168.0.107:50329    157.240.20.15:443      ESTABLISHED     10836
TCP    192.168.0.107:50331    111.221.29.74:443      ESTABLISHED     10072
TCP    192.168.0.107:50332    162.125.18.133:443     ESTABLISHED     9668
TCP    192.168.0.107:50351    40.77.226.194:443      ESTABLISHED     8620
TCP    192.168.0.107:50460    66.102.1.189:443       ESTABLISHED     10836
TCP    192.168.0.107:50470    66.102.1.189:443       ESTABLISHED     10836
TCP    192.168.0.107:50501    192.30.253.125:443     ESTABLISHED     11184
TCP    192.168.0.107:50513    40.77.226.194:443      ESTABLISHED     8620
TCP    192.168.0.107:50529    87.98.218.198:443      ESTABLISHED     12540
TCP    192.168.0.107:50530    172.217.21.46:443      ESTABLISHED     10836
TCP    192.168.0.107:50616    172.217.21.46:443      ESTABLISHED     10836
TCP    192.168.0.107:50630    162.125.18.133:443     ESTABLISHED     9668
TCP    192.168.0.107:50641    172.217.21.37:443      ESTABLISHED     10836
TCP    192.168.0.107:50645    162.125.66.4:443       CLOSE_WAIT      9668
TCP    192.168.0.107:50668    87.98.218.198:443      ESTABLISHED     12540
TCP    192.168.0.107:50703    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50709    192.30.253.125:443     ESTABLISHED     11184
TCP    192.168.0.107:50744    87.98.218.198:443      ESTABLISHED     12540
TCP    192.168.0.107:50828    162.125.66.3:443       CLOSE_WAIT      9668
TCP    192.168.0.107:50830    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50831    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50832    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50834    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50835    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50836    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50837    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50839    192.30.253.125:443     ESTABLISHED     11184
TCP    192.168.0.107:50844    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50847    192.30.253.124:443     ESTABLISHED     11184
TCP    192.168.0.107:50857    192.30.253.124:443     ESTABLISHED     11184
TCP    192.168.0.107:50863    162.125.34.137:443     CLOSE_WAIT      9668
TCP    192.168.0.107:50865    172.217.21.46:443      TIME_WAIT       0
TCP    192.168.0.107:50866    172.217.21.46:443      ESTABLISHED     10836
TCP    192.168.0.107:50910    35.186.213.138:443     TIME_WAIT       0
TCP    192.168.0.107:50923    172.217.21.46:443      ESTABLISHED     10836
TCP    192.168.0.107:50925    40.117.190.72:443      ESTABLISHED     4040
TCP    192.168.0.107:50927    172.217.21.42:443      ESTABLISHED     11184
TCP    192.168.0.107:50949    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50950    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50951    151.101.0.133:443      ESTABLISHED     11184
TCP    192.168.0.107:50954    192.30.253.124:443     ESTABLISHED     11184
TCP    192.168.0.107:50959    40.117.190.72:443      TIME_WAIT       0
TCP    192.168.0.107:50969    192.30.253.113:22      TIME_WAIT       0
TCP    192.168.0.107:50978    87.98.218.198:443      ESTABLISHED     12540
TCP    192.168.0.107:50984    151.101.1.69:443       ESTABLISHED     11184
TCP    192.168.0.107:50985    192.0.73.2:443         ESTABLISHED     11184
TCP    192.168.0.107:50986    104.16.112.18:443      ESTABLISHED     11184
TCP    192.168.0.107:50991    198.252.206.25:443     ESTABLISHED     11184
TCP    192.168.0.107:50993    192.168.0.10:3910      TIME_WAIT       0
TCP    192.168.0.107:50994    192.168.0.10:3910      TIME_WAIT       0
TCP    192.168.0.107:50997    23.210.254.37:443      ESTABLISHED     912
TCP    192.168.0.107:50998    23.210.254.37:443      ESTABLISHED     912
TCP    192.168.0.107:50999    23.210.254.37:443      ESTABLISHED     912
TCP    192.168.0.107:51001    23.210.254.37:443      ESTABLISHED     912
TCP    192.168.0.107:51006    40.117.190.72:443      ESTABLISHED     11992
TCP    192.168.0.107:51008    40.69.218.62:443       ESTABLISHED     7056
TCP    192.168.0.107:51010    172.217.21.46:443      ESTABLISHED     11184
TCP    [::]:135               [::]:0                 LISTENING       1152
TCP    [::]:445               [::]:0                 LISTENING       4
TCP    [::]:2179              [::]:0                 LISTENING       4696
TCP    [::]:5357              [::]:0                 LISTENING       4
TCP    [::]:7680              [::]:0                 LISTENING       7056
TCP    [::]:17500             [::]:0                 LISTENING       9668
TCP    [::]:49664             [::]:0                 LISTENING       784
TCP    [::]:49665             [::]:0                 LISTENING       1628
TCP    [::]:49666             [::]:0                 LISTENING       2028
TCP    [::]:49667             [::]:0                 LISTENING       3560
TCP    [::]:49800             [::]:0                 LISTENING       856
TCP    [::]:49821             [::]:0                 LISTENING       892
TCP    [::1]:5432             [::]:0                 LISTENING       11100
TCP    [::1]:50000            [::]:0                 LISTENING       11100
TCP    [::1]:50001            [::]:0                 LISTENING       11100
TCP    [::1]:51000            [::]:0                 LISTENING       11100
UDP    0.0.0.0:53             *:*                                    5620
UDP    0.0.0.0:3702           *:*                                    2084
UDP    0.0.0.0:3702           *:*                                    2084
UDP    0.0.0.0:5050           *:*                                    6616
UDP    0.0.0.0:5353           *:*                                    11184
UDP    0.0.0.0:5353           *:*                                    11184
UDP    0.0.0.0:5353           *:*                                    11184
UDP    0.0.0.0:5353           *:*                                    3080
UDP    0.0.0.0:5353           *:*                                    11184
UDP    0.0.0.0:5353           *:*                                    11184
UDP    0.0.0.0:5353           *:*                                    11184
UDP    0.0.0.0:5353           *:*                                    11184
UDP    0.0.0.0:5355           *:*                                    3080
UDP    0.0.0.0:17500          *:*                                    9668
UDP    0.0.0.0:49670          *:*                                    2084
UDP    0.0.0.0:57329          *:*                                    5620
UDP    0.0.0.0:57330          *:*                                    5620
UDP    0.0.0.0:59529          *:*                                    8620
UDP    0.0.0.0:60605          *:*                                    11184
UDP    10.0.75.1:137          *:*                                    4
UDP    10.0.75.1:138          *:*                                    4
UDP    10.0.75.1:1900         *:*                                    2620
UDP    10.0.75.1:61326        *:*                                    2620
UDP    127.0.0.1:1900         *:*                                    2620
UDP    127.0.0.1:60816        *:*                                    4616
UDP    127.0.0.1:61328        *:*                                    2620
UDP    169.254.105.83:137     *:*                                    4
UDP    169.254.105.83:138     *:*                                    4
UDP    169.254.105.83:1900    *:*                                    2620
UDP    169.254.105.83:61330   *:*                                    2620
UDP    172.30.146.241:67      *:*                                    5620
UDP    172.30.146.241:68      *:*                                    5620
UDP    172.30.146.241:1900    *:*                                    2620
UDP    172.30.146.241:61329   *:*                                    2620
UDP    192.168.0.107:137      *:*                                    4
UDP    192.168.0.107:138      *:*                                    4
UDP    192.168.0.107:1900     *:*                                    2620
UDP    192.168.0.107:61327    *:*                                    2620
UDP    [::]:3702              *:*                                    2084
UDP    [::]:3702              *:*                                    2084
UDP    [::]:5353              *:*                                    11184
UDP    [::]:5353              *:*                                    11184
UDP    [::]:5353              *:*                                    3080
UDP    [::]:5353              *:*                                    11184
UDP    [::]:5355              *:*                                    3080
UDP    [::]:49671             *:*                                    2084
UDP    [::]:57331             *:*                                    5620
UDP    [::]:59529             *:*                                    8620
UDP    [::1]:1900             *:*                                    2620
UDP    [::1]:61323            *:*                                    2620
UDP    [fe80::30eb:ad8f:f94a:b774%26]:1900  *:*                                    2620
UDP    [fe80::30eb:ad8f:f94a:b774%26]:61324  *:*                                    2620
UDP    [fe80::718c:22bb:fd97:c06c%23]:1900  *:*                                    2620
UDP    [fe80::718c:22bb:fd97:c06c%23]:61322  *:*                                    2620
UDP    [fe80::85d0:3b5c:7746:6953%5]:1900  *:*                                    2620
UDP    [fe80::85d0:3b5c:7746:6953%5]:61325  *:*                                    2620

The f# code I used to test for open ports:

open System.Net
open System.Net.Sockets

let ipAddress = IPAddress([| (byte)0; (byte)0; (byte)0; (byte)0 |])

let ipEndpoint portNumber = (IPEndPoint(ipAddress, portNumber), portNumber)

let getPorts = seq { for i in 1 .. 65535 -> i }

let checkIfPortAvailable (endpoint, portNumber) =
    use listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)
    try
        listener.Bind(endpoint)
        (portNumber, true, null)
    with
        | ex -> (portNumber, false, ex)

[<EntryPoint>]
let main argv =
    getPorts
    |> Seq.map ipEndpoint
    |> Seq.map checkIfPortAvailable
    |> Seq.where (fun (_, works, _) -> not works)
    |> Seq.where (fun (_, _, ex) -> ex.Message.Contains("An attempt was made to access a socket in a way forbidden by its access permissions"))
    |> Seq.iteri (fun index (port, _, _) -> printfn "%5d: %d" index port)

    0

and the output when executing the application (it was executed right after the netstat command):

0: 445
1: 5357
2: 7680
3: 49670
4: 49671
5: 49672
6: 49673
7: 49674
8: 49675
9: 49676
10: 49677
11: 49678
12: 49679
13: 49680
14: 49681
15: 49682
16: 49683
17: 49684
18: 49685
19: 49686
20: 49687
21: 49688
22: 49689
23: 49710
24: 49711
25: 49712
26: 49713
27: 49714
28: 49715
29: 49716
30: 49717
31: 49718
32: 49719
33: 49720
34: 49721
35: 49722
36: 49723
37: 49724
38: 49725
39: 49726
40: 49727
41: 49728
42: 49729
43: 49730
44: 49731
45: 49732
46: 49733
47: 49734
48: 49735
49: 49736
50: 49737
51: 49738
52: 49739
53: 49740
54: 49741
55: 49742
56: 49743
57: 49744
58: 49745
59: 49746
60: 49747
61: 49748
62: 49749
63: 49750
64: 49751
65: 49752
66: 49753
67: 49754
68: 49755
69: 49756
70: 49757
71: 49758
72: 49759
73: 49770
74: 49771
75: 49772
76: 49773
77: 49774
78: 49775
79: 49776
80: 49777
81: 49778
82: 49779
83: 49780
84: 49781
85: 49782
86: 49783
87: 49784
88: 49785
89: 49786
90: 49787
91: 49788
92: 49789
93: 49790
94: 49791
95: 49792
96: 49793
97: 49794
98: 49795
99: 49796
100: 49797
101: 49798
102: 49799
103: 49825
104: 49826
105: 49827
106: 49828
107: 49829
108: 49830
109: 49831
110: 49832
111: 49833
112: 49834
113: 49835
114: 49836
115: 49837
116: 49838
117: 49839
118: 49840
119: 49841
120: 49842
121: 49843
122: 49844
123: 49845
124: 49846
125: 49847
126: 49848
127: 49849
128: 49850
129: 49851
130: 49852
131: 49853
132: 49854
133: 49855
134: 49856
135: 49857
136: 49858
137: 49859
138: 49860
139: 49861
140: 49862
141: 49863
142: 49864
143: 49866
144: 49867
145: 49868
146: 49869
147: 49870
148: 49871
149: 49872
150: 49873
151: 49874
152: 49875
153: 49876
154: 49877
155: 49878
156: 49879
157: 49880
158: 49881
159: 49882
160: 49883
161: 49884
162: 49885
163: 50274
164: 50275
165: 50276
166: 50277
167: 50278
168: 50279
169: 50280
170: 50281
171: 50282
172: 50283
173: 50284
174: 50285
175: 50286
176: 50287
177: 50288
178: 50289
179: 50290
180: 50291
181: 50292
182: 50293

The port which I'd actually want to have opened is 49786 but the amount of ports reported with permission denied by my small testing algorithm, but not allocated when checking netstat is confusing me

UPDATE 3:

Now that simply running shutdown /s /f /t 0 doesn't work anymore, I've found another workaround which seems to be working fairly reliably in https://social.technet.microsoft.com/Forums/windows/en-US/e0cc1b3e-2c63-451a-abc8-8e1e66a23287/an-attempt-was-made-to-access-a-socket-in-a-way-forbidden-by-its-access-permissions?forum=w7itpronetworking - netsh int ip reset and restarting the computer (normal windows 10 restart is enough) and the port was consistently available on the next start so far (but it was just 3 tries)

UPDATE 4:

Summary of the comments:

  • setting SO_REUSEADDR in the small f# script makes things only worse
  • trying to close skype, VPN applications, dropbox and similar didn't have any effect (yes I actually made sure that the processes are terminated)

Solution 1:

The reason is that Hyper-V takes over these ports, to prevent it from happening do the following:

  1. dism.exe /Online /Disable-Feature:Microsoft-Hyper-V (will have to restart)
  2. netsh int ipv4 add excludedportrange protocol=tcp startport=<your port> numberofports=1
  3. dism.exe /Online /Enable-Feature:Microsoft-Hyper-V /All

The original solution is here.

If after that you still can't bind to the port do the following:

  • netsh http add iplisten 0.0.0.0 for IPv4
  • netsh http add iplisten :: for IPv6

More information is here.

Solution 2:

A faulty Windows upgrade is the reason for the problems with reserved tcp ports. See https://github.com/docker/for-win/issues/3171#issuecomment-554587817

With this the problem should be fixed: netsh int ipv4 set dynamic tcp start=49152 num=16384

That would explain why a clean install fixes these type of problems.

Solution 3:

I also had the Hyper-v/WSL problem above, but found a simpler solution:

I had stopped docker before doing this, not sure if it's necessary.

Run as admin:

net stop winnat
net start winnat