I have a TailFile -> PublishKafka processor in NiFi for squid logs.
But when I push the logs to Kafka, they don't appear in Elasticsearch. They do show up in Kafka when I run the ./kafka-console-producer.sh script.
Storm also sees the new topic I created in Kafka named "squid"
But Elasticsearch/Kibana don't see any of the logs.
Is Metron not receiving the logs from Kafka? When I check the sensor in Metron Management UI, it says the Kafka Topic is "emitting" so I assume that Metron does see the logs coming into Kafka?
I am not sure why I don't see the logs in Kibana or when I curl -xget $ELASTICSEARCH:9200
My Kafka Broker, Elasticsearch, Kibana, Metron, Zookeeper are all on the same physical node.
Can someone please assist with this?