I wrote a program to create squid log (about 800,000) and sent to Metron by NiFi.
My data was sent correctly into kafka because I count kafka-log and it was exactly 800,000 and in storm I saw exactly this number but in the HDFS and Elasticsearch l saw less than 800000 (about 300,000).
I didn't see any error in the Storm or Elasticsearch's log.
I started Bro for 30 minutes and count the number of log in the bro, kafka-log, Strom, HDFS and Elasticsearch. I understood the number of data in HDFS and Elasticsearch is less than Storm and kafka-log.