Kafka consumer list

I need to find out a way to ask Kafka for a list of topics. I know I can do that using the kafka-topics.sh script included in the bin\ directory. Once I have this list, I need all the consumers per topic. I could not find a script in that directory, nor a class in the kafka-consumer-api library that allows me to do it.

The reason behind this is that I need to figure out the difference between the topic's offset and the consumers' offsets.

Is there a way to achieve this? Or do I need to implement this functionality in each of my consumers?

Use kafka-consumer-groups.sh

For example

bin/kafka-consumer-groups.sh  --list --bootstrap-server localhost:9092

bin/kafka-consumer-groups.sh --describe --group mygroup --bootstrap-server localhost:9092

you can use this for 0.9.0.0. version kafka

./kafka-consumer-groups.sh --list --zookeeper hostname:potnumber

to view the groups you have created. This will display all the consumer group names.

 ./kafka-consumer-groups.sh --describe --zookeeper hostname:potnumber  --describe  --group consumer_group_name

To view the details

GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER

I realize that this question is nearly 4 years old now. Much has changed in Kafka since then. This is mentioned above, but only in small print, so I write this for users who stumble over this question as late as I did.

1. Offsets by default are now stored in a Kafka Topic (not in Zookeeper any more), see Offsets stored in Zookeeper or Kafka?
2. There's a kafka-consumer-groups utility which returns all the information, including the offset of the topic and partition, of the consumer, and even the lag (Remark: When you ask for the topic's offset, I assume that you mean the offsets of the partitions of the topic). In my Kafka 2.0 test cluster:

kafka-consumer-groups --bootstrap-server kafka:9092 --describe
    --group console-consumer-69763 Consumer group 'console-consumer-69763' has no active members.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
pytest          0          5               6               1               -               -               -
``

All the consumers per topic

(Replace --zookeeper with --bootstrap-server to get groups stored by newer Kafka clients)

Get all consumers-per-topic as a table of topictabconsumer:

for t in `kafka-consumer-groups.sh --zookeeper <HOST>:2181 --list 2>/dev/null`; do
    echo $t | xargs -I {} sh -c "kafka-consumer-groups.sh --zookeeper <HOST>:2181 --describe --group {} 2>/dev/null | grep ^{} | awk '{print \$2\"\t\"\$1}' "
done > topic-consumer.txt

Make this pairs unique:

cat topic-consumer.txt | sort -u > topic-consumer-u.txt

Get the desired one:

less topic-consumer-u.txt | grep -i <TOPIC>