you have to fiil in nifi.remote.input.host= at conf/nifi.properties. this work when you deploy nifi to actual host  ... View more

ok changed to 2 secs. Will keep an eye on it to monitor for future data duplication. Going to accept this as answer for now. Thank you. ... View more

I stand corrected. It seems if you are doing Tailfile in a cluster, the directories and files must exist on all hosts. ... View more

Thanks that was a great answer. ... View more

@Eric Lloyd, for this case make we cannot mention wild cards like [*] as this processor wont accepting those regex. change FilestoTail property to test[1-2]/[\d|a-z.*]{1,}/test.log Expression Explanation:- test[1-2] --look for test1 or test2 Then property expression check for [\d|a-z.*]{1,} --check for directory name as having digits or letters more than one time and lists all those directories recursively. Configs:- For your case:- Files toTail property should be some thing like below versions/[\d|a-z.*]{1,}/<your-log-file-name> ... View more

Update: I had 4 flows running at the same time using these Zookeeper instances. When I reduced them from 4 to 2, Zookeeper no longer crashes. ... View more

@Eric Lloyd You may want to look in to using the SplitRecord processor instead of SplitContent. You could use a the GrokReader to split you log input. Here is a great article that includes a sample grok pattern for nifi log's format: - https://community.hortonworks.com/articles/131320/using-partitionrecord-grokreaderjsonwriter-to-pars.html - Thank you, Matt ... View more

Wow the solution to remove the extra lines was creating a giant bottleneck... Hmm ... View more

Awesome thanks again ! ... View more

Just my Two Cents on it, If you are experiencing a lot of queued flow-files and you want to clear them without having to reload the config.yml, a very quick and dirty way would be to clear all the repos on the minifi edge. Here how i do it - assuming your minifi sits in /opt - you will lose all queued data by doing this, but you no longer have queues.(not ideal) -- before cleanup /opt/minifi/bin/minifi.sh flowStatus connection:all:health,stats | tail -2 | head -1| jq '.' /opt/minifi/bin/minifi.sh stop rm -rf /opt/minifi/flowfile_repository/* rm -rf /opt/minifi/content_repository/* rm -rf /opt/minifi/provenance_repository/* /opt/minifi/bin/minifi.sh start -- after cleanup /opt/minifi/bin/minifi.sh flowStatus connection:all:health,stats | tail -2 | head -1| jq '.' Ideal solution: Use the minifi.sh flowStatus cmd to push data into the NiFi server and monitor this types of stuff. From here you can develop nifi flows to act on what needs to be done to those edge minifi. you can even build a dashboard from this data and actually see how to edges are doing. No doubt Horton has this on the to do list. ... View more