Support Questions
Find answers, ask questions, and share your expertise

HDFS and Elasticsearch data is less than Storm

HDFS and Elasticsearch data is less than Storm

Explorer

Hi

I use Metron 0.4.1 and HDP 2.5.0.0.

I wrote a program to create squid log (about 800,000) and sent to Metron by NiFi.

My data was sent correctly into kafka because I count kafka-log and it was exactly 800,000 and in storm I saw exactly this number but in the HDFS and Elasticsearch l saw less than 800000 (about 300,000).

I didn't see any error in the Storm or Elasticsearch's log.

I started Bro for 30 minutes and count the number of log in the bro, kafka-log, Strom, HDFS and Elasticsearch. I understood the number of data in HDFS and Elasticsearch is less than Storm and kafka-log.

How can I solve it?

Thank you.