Why I cannot connect to Kafka from outside?

I am running kafka on ec2 instance. So amazon ec2 instance has two ips one is internal ip and second one is for external use.

I created producer from local machine, but it redirect to internal ip and give me connection unsuccessful error. Can anybody help me to configure kafka on ec2 instance, so that I can run producer from local machine. I am tried many combinations but didn't work.

Solution 1:

In the Kafka FAQ (updated for new properties) you can read:

When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in bootstrap.servers in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress(). Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip and port to be registered in ZK by setting the advertised.listeners property in server.properties.

Solution 2:

I solved this problem, by setting advertised.host.name in server.properties and metadata.broker.list in producer.properties to public IP address and host.name to 0.0.0.0.

Solution 3:

The easiest way how to reach your Kafka server (version kafka_2.11-1.0.0) on EC2 from consumer in external network is to change the properties file

kafka_2.11-1.0.0/config/server.properties

And modify the following line

listeners=PLAINTEXT://ec2-XXX-XXX-XXX-XXX.eu-central-1.compute.amazonaws.com:9092

Using your public address

Verified on 2.11-2.0.0

Solution 4:

I just did this in AWS. First get the Kafka server to listen on the correct interface/IP using host.name. For your case this would be the internal IP, not localhost, since your intent is for outside Kafka clients to connect. Any local clients will need to use that same address, not localhost.

Then set advertised.host.name to a host name, not an IP address. The trick is to get that host name to always resolve to the correct IP for both internal and external machines. I use /etc/hosts inside and DNS outside. See my full answer about Kafka and name resolution here.

Solution 5:

If you want to access from LAN, change following 2 files-

1. In config/server.properties:

   advertised.listeners=PLAINTEXT://server.ip.in.lan:9092
   

2. In config/producer.properties:

   bootstrap.servers=server.ip.in.lan:9092
   

In my case, the server.ip.in.lan value was 192.168.15.150