Support Questions
Find answers, ask questions, and share your expertise

Unable to see records in Kafka topic

Super Guru

Team

I am trying to the truck iot app. Everything is setup and apparently running except that I don't see any processing in Streaming Analytics Manager. This is not surprising because when I try to check my Kafka topic using "kafka-console-consumer", I get nothing:

root@mqureshi4 bin]# ./kafka-console-consumer.sh --bootstrap-server localhost:6667 --topic truck_speed_events_avro --from-beginning

Nifi shows thousands of records have been sent. Data Provenance in Nifi shows records being "sent" by Publish Kafka Processor. What could be wrong here? I have verified and there are no typos in topic names.

3 REPLIES 3

Re: Unable to see records in Kafka topic

Super Guru

Actually, I am able to see my data in Kafka. its under /kafka-logs" folder as configured in Kafka settings in Ambari. But my zookeeper directory seems almost emopty. I have only the following:

[root@mqureshi4 truck_events_avro-0]# ls -l /hadoop/zookeeper/

total 4

-rw-r--r-- 1 root      root    1 Mar 16 15:50 myid

drwxr-xr-x 2 zookeeper hadoop 18 Mar 16 15:50 version-2

[root@mqureshi4 truck_events_avro-0]# ls -l /hadoop/zookeeper/version-2/

total 22064

-rw-r--r-- 1 zookeeper hadoop 67108880 Mar 20 18:29 log.1

I don't see a Kafka node here. I am wondering if that's the issue.

Re: Unable to see records in Kafka topic

Rising Star

So me see I understand:

  • You are running the simulator and data is being generated
  • You have nifi up and running and it is ingesting the data being simulator and pushing that data into a Kafka topic
  • You have verified in Ambari/Grafana and using kafka-consumer that data is actually being sent to the kafka topic successfully? Please verify by sending a screenshot.
  • You have SAM app that is deployed and it is supposed to read from the kafka topic. But you are not seeing any events being read form the Kafka spout? Please send screenshot of the SAM running app and the the ambari storm view of tthe topology that is being deployed

Questions:

  • How are you verifying that SAM is not picking up any tuples?
  • Did you verify that the topology was deployed successfully without errors in Ambari using the new Storm Ambari View?

Re: Unable to see records in Kafka topic

Super Guru

This seems to be an issue. Perhaps case sensitive? I have this issue as well. Kafka messages not picked up by SAM.