kafka consumer not showing the consumed messages ?

[#1]

I am following https://kafka.apache.org/quickstart and few weeks earlier everything was working well, but when I started it over again today is not working, I am using kafka_2.11-1.1.0.

I created the new topic 'rahul' with the following command :

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic rahul
Created topic "rahul"

I also re-checked the topics with

bin/kafka-topics.sh --list --zookeeper localhost:2181

`__consumer_offsets
rahhy
rahul`

started the producer:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic rahul
hey

>hi
>hello

But when the time comes to consumer to show the messages: there is nothing although the command seems to be executed, but no messages.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic rahul --from-beginning

I even tried consuming both ways as zookeeper and bootstrap server, but none of them are giving messages published.

[#2]

Hey @Rahul Kumar!
How much is set for log.retention.hours?
And could you check if your kafka-console-consumer are creating a consumer group?

[root@node1 ~]# kafka-consumer-groups.sh --zookeeper $ZKENSEMBLE --list
#Or
[root@node1 ~]# kafka-consumer-groups.sh --bootstrap-server node1:6667 --list 
#If shows smtg, try to describe it with the --group <consumer-group-mynumber> and --describe

#And we can also check the offset of your topic.

root@node1 ~]# kafka-run-class.sh kafka.tools.ExportZkOffsets --zkconnect $ZKENSEMBLE --output-file zk_offset_kafka
[root@node1 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list node1:6667 --topic vin-hcc-nifi --time -1 #latest
vin-hcc-nifi:2:1
vin-hcc-nifi:1:1
vin-hcc-nifi:0:1
[root@node1 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list node1:6667 --topic vin-hcc-nifi --time -2 #earliest 
vin-hcc-nifi:2:0
vin-hcc-nifi:1:0
vin-hcc-nifi:0:0 

Hope this helps!

[#3]

Hi @Vinicius Higa Murakami

I removed zookeeper,kafka and their respective log directories.
I installed zookeeper and kafka again.

So for the first time messages were consumed and published too on the terminal interface.
After that I stopped kafka and restarted kafka again... from this point the message is not getting consumed.

Why this is happening ? This is very unusal from kafka 😞

[#4]

Hey @Rahul Kumar.
Just asking, but after you stopped kafka/zookeeper, did you tried to produce and consume messages again?
For example, let's say that you just did a kafka-console-consumer after 7 days, probably you won't be able to see that messages again on that topic, because Kafka has a parameter that retains messages for a determined period of time, which is log.retention.hours = 168 hours (7 days) by default (you can change it).

But, if you did the whole process again (create a topic, kafka-console-producer and kafka-console-consumer) after the kafka cluster was down, then we may need to take a look at the errors from the logs of Kafka/ZK and watch the consumer groups/offsets.
Hope this helps!

[#5]

Hello @Rahul Kumar. Could you check your /etc/hosts? I saw you are using "localhost" as your broker/bootstrap. Try to change it to your "hostname" instead of "localhost".

Some softwares doesn´t use "loopback" as a listener port by default.

Besides that, if you are using the native Kafka from HortonWorks, the default port for brokers is 6667 or 6668 (when using Kerberos) and not 9092.

I hope this works.