Support Questions
Find answers, ask questions, and share your expertise

Elasticsearch not recieving data from Kafka or Metron

Elasticsearch not recieving data from Kafka or Metron

Explorer

I have a TailFile -> PublishKafka processor in NiFi for squid logs.

But when I push the logs to Kafka, they don't appear in Elasticsearch. They do show up in Kafka when I run the ./kafka-console-producer.sh script.

Storm also sees the new topic I created in Kafka named "squid"

But Elasticsearch/Kibana don't see any of the logs.

 

Is Metron not receiving the logs from Kafka? When I check the sensor in Metron Management UI, it says the Kafka Topic is "emitting" so I assume that Metron does see the logs coming into Kafka?

 

I am not sure why I don't see the logs in Kibana or when I curl -xget $ELASTICSEARCH:9200

 

My Kafka Broker, Elasticsearch, Kibana, Metron, Zookeeper are all on the same physical node.

Can someone please assist with this?