Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

kafka consumer not showing the consumed messages ?

Contributor

I am following https://kafka.apache.org/quickstart and few weeks earlier everything was working well, but when I started it over again today is not working, I am using kafka_2.11-1.1.0.

I created the new topic 'rahul' with the following command :

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic rahul
Created topic "rahul"

I also re-checked the topics with

bin/kafka-topics.sh --list --zookeeper localhost:2181

`__consumer_offsets
rahhy
rahul`

started the producer:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic rahul
hey

>hi
>hello

But when the time comes to consumer to show the messages: there is nothing although the command seems to be executed, but no messages.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic rahul --from-beginning

I even tried consuming both ways as zookeeper and bootstrap server, but none of them are giving messages published.

76617-no-msg-in-consumer.png

4 REPLIES 4

Hey @Rahul Kumar!
How much is set for log.retention.hours?
And could you check if your kafka-console-consumer are creating a consumer group?

[root@node1 ~]# kafka-consumer-groups.sh --zookeeper $ZKENSEMBLE --list
#Or
[root@node1 ~]# kafka-consumer-groups.sh --bootstrap-server node1:6667 --list
#If shows smtg, try to describe it with the --group <consumer-group-mynumber> and --describe
#And we can also check the offset of your topic.
root@node1 ~]# kafka-run-class.sh kafka.tools.ExportZkOffsets --zkconnect $ZKENSEMBLE --output-file zk_offset_kafka
[root@node1 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list node1:6667 --topic vin-hcc-nifi --time -1 #latest
vin-hcc-nifi:2:1
vin-hcc-nifi:1:1
vin-hcc-nifi:0:1
[root@node1 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list node1:6667 --topic vin-hcc-nifi --time -2 #earliest 
vin-hcc-nifi:2:0
vin-hcc-nifi:1:0
vin-hcc-nifi:0:0 

Hope this helps!

Contributor

Hi @Vinicius Higa Murakami

I removed zookeeper,kafka and their respective log directories.
I installed zookeeper and kafka again.

So for the first time messages were consumed and published too on the terminal interface.
After that I stopped kafka and restarted kafka again... from this point the message is not getting consumed.

Why this is happening ? This is very unusal from kafka 😞

Hey @Rahul Kumar.
Just asking, but after you stopped kafka/zookeeper, did you tried to produce and consume messages again?
For example, let's say that you just did a kafka-console-consumer after 7 days, probably you won't be able to see that messages again on that topic, because Kafka has a parameter that retains messages for a determined period of time, which is log.retention.hours = 168 hours (7 days) by default (you can change it).

But, if you did the whole process again (create a topic, kafka-console-producer and kafka-console-consumer) after the kafka cluster was down, then we may need to take a look at the errors from the logs of Kafka/ZK and watch the consumer groups/offsets.
Hope this helps!

Explorer

Hello @Rahul Kumar. Could you check your /etc/hosts? I saw you are using "localhost" as your broker/bootstrap. Try to change it to your "hostname" instead of "localhost".

Some softwares doesn´t use "loopback" as a listener port by default.

Besides that, if you are using the native Kafka from HortonWorks, the default port for brokers is 6667 or 6668 (when using Kerberos) and not 9092.

I hope this works.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.