nginx reverse-proxy 502 (111: connection refused) while connecting to upstream [duplicate]

This is a Canonical Question about Connection Refused

We see a lot of questions to the effect

When I try to connect to a system I get a message

Connection refused

Why is this ?

Solution 1:

Note: This message is a symptom of the problem you are trying to solve. Understanding the cause of the message will ultimately lead you to solving your problem.

The message 'Connection Refused' has two main causes:

Nothing is listening on the IP:Port you are trying to connect to.
The port is blocked by a firewall.

No process is listening.

This is by far the most common reason for the message. First ensure that you are trying to connect to the correct system. If you are then to determine if this is the problem, on the remote system run netstat or ss¹ e.g. if you are expecting a process to be listening on port 22222

sudo netstat -tnlp | grep :22222

ss -tnlp | grep :22222

For OSX a suitable command is

sudo netstat -tnlp tcp | grep '\.80 '

If nothing is listening then the above will produce no output. If you see some output then confirm that it's what you expect then see the firewall section below.

If you don't have access to the remote system and want to confirm the problem before reporting it to the relevant administrators you can use tcpdump (wireshark or similar).

When a connection is attempted to an IP:port where nothing is listening, the response from the remote system to the initial SYN packet is a packet with the flags RST,ACK set. This closes the connection and causes the Connection Refused message e.g.

$ sudo tcpdump -n host 192.0.2.1 and port 22222
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp14s0, link-type EN10MB (Ethernet), capture size 262144 bytes

12:31:27.013976 IP 192.0.2.2.34390 > 192.0.2.1.22222: Flags [S], seq 1207858804, win 29200, options [mss 1460,sackOK,TS val 15306344 ecr 0,nop,wscale 7], length 0

12:31:27.020162 IP 192.0.2.1.22222 > 192.0.2.2.34390: Flags [R.], seq 0, ack 1207858805, win 0, length 0

Note that tcpdump uses a . to represent the ACK flag.

Port is blocked by a firewall

If the port is blocked by a firewall and the firewall has been configured to respond with icmp-port-unreachable this will also cause a connection refused message. Again you can see this with tcpdump (or similar)

$ sudo tcpdump -n icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on enp14s0, link-type EN10MB (Ethernet), capture size 262144 bytes 13:03:24.149897 IP 192.0.2.1 > 192.0.2.2: ICMP 192.0.2.1 tcp port 22222 unreachable, length 68

Note that this also tells us where the blocking firewall is.

So now you know what's causing the Connection refused message you should take appropriate action e.g. contact the firewall administrator or investigate the reason for the process not listening.

¹ Other tools are likely available.