I am following https://kafka.apache.org/quickstart and few weeks earlier everything was working well, but when I started it over again today is not working, I am using kafka_2.11-1.1.0.
I created the new topic 'rahul' with the following command :
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic rahul Created topic "rahul"
I also re-checked the topics with
bin/kafka-topics.sh --list --zookeeper localhost:2181
started the producer:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic rahul hey
But when the time comes to consumer to show the messages: there is nothing although the command seems to be executed, but no messages.
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic rahul --from-beginning
I even tried consuming both ways as zookeeper and bootstrap server, but none of them are giving messages published.
Hey @Rahul Kumar!
How much is set for log.retention.hours?
And could you check if your kafka-console-consumer are creating a consumer group?
[root@node1 ~]# kafka-consumer-groups.sh --zookeeper $ZKENSEMBLE --list #Or#And we can also check the offset of your topic.
[root@node1 ~]# kafka-consumer-groups.sh --bootstrap-server node1:6667 --list
#If shows smtg, try to describe it with the --group <consumer-group-mynumber> and --describe
root@node1 ~]# kafka-run-class.sh kafka.tools.ExportZkOffsets --zkconnect $ZKENSEMBLE --output-file zk_offset_kafka [root@node1 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list node1:6667 --topic vin-hcc-nifi --time -1 #latest vin-hcc-nifi:2:1 vin-hcc-nifi:1:1 vin-hcc-nifi:0:1 [root@node1 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list node1:6667 --topic vin-hcc-nifi --time -2 #earliest vin-hcc-nifi:2:0 vin-hcc-nifi:1:0 vin-hcc-nifi:0:0
Hope this helps!
Hi @Vinicius Higa Murakami
I removed zookeeper,kafka and their respective log directories.
I installed zookeeper and kafka again.
So for the first time messages were consumed and published too on the terminal interface.
After that I stopped kafka and restarted kafka again... from this point the message is not getting consumed.
Why this is happening ? This is very unusal from kafka :(
Hey @Rahul Kumar.
Just asking, but after you stopped kafka/zookeeper, did you tried to produce and consume messages again?
For example, let's say that you just did a kafka-console-consumer after 7 days, probably you won't be able to see that messages again on that topic, because Kafka has a parameter that retains messages for a determined period of time, which is log.retention.hours = 168 hours (7 days) by default (you can change it).
But, if you did the whole process again (create a topic, kafka-console-producer and kafka-console-consumer) after the kafka cluster was down, then we may need to take a look at the errors from the logs of Kafka/ZK and watch the consumer groups/offsets.
Hope this helps!
Hello @Rahul Kumar. Could you check your /etc/hosts? I saw you are using "localhost" as your broker/bootstrap. Try to change it to your "hostname" instead of "localhost".
Some softwares doesn´t use "loopback" as a listener port by default.
Besides that, if you are using the native Kafka from HortonWorks, the default port for brokers is 6667 or 6668 (when using Kerberos) and not 9092.
I hope this works.