I have an unusual issue. I am am using both Flume and Nifi to process data in Hadoop and then read those HDFS files with HUNK (Splunk for Hadoop Analytics). When I run a query on the data ingested through Flume, it is read quickly (about 100,000+ events per second). ** Lately ** when I run a query on the data ingested by Nifi, it is slow, processing about 1000-2000 events per second. I say **lately** because just two days ago, the query of the Nifi ingested data was quick as the Flume data. 😮 Whaa? Some specifics: Flume: folders in HDFS separated and configured by Year/Month/Day/(one single log file for the whole day) Nifi: folders in HDFS separated and configured by Year/Month/Day/Hour/(many log files, about 13KB each - done this way in order to prevent possible data corruption of a cessation of data ingestion through Nifi while a data is being appended to a single file such as in Flume) I'd say it was the difference in file structure except: (1) the more granular file directory structure of the Nifi ingested files are suppose to produce FASTER searches in HUNK (2) As I said, just two days ago, the queries being run on this Nifi ingested data was just as fast as the data being ingested by Flume. I can provide more info but Im looking for some ideas on what might be causing this or some ideas on how to troubleshoot such a problem. ... View more

For some reason even though my Kafka version is listed as kafka_2.10-0.10.0.2.5.0.0-1245.jar (which I am pretty sure is Kafka 0.10), the only processor that works for me is GetKafka. Consume_Kafka and Consume_Kafka_0_10 both will hang and not process any data... if I stop the nifi flow it does give an error seen in the screenshot. I have left it running up to 10 mins waiting for it to connect but it never seems to. I have also including a screenshot of my KafkaConsume properties window which is using the same values as the ones I put in GetKafka when it works. ... View more