The issue about waiting for metadata likely means the machine running NiFi can't reach one of the Kafka brokers hosting the topic. There is some property in Kafka that I can't remember that controls what host/ip it presents to external clients which sometimes needs to be set. ... View more

You can, but you are just shifting the problem downstream from ListenTCP to SplitText. SplitText now has to produce thousands/millions of flow files that would have been coming out of ListenTCP. It is slightly better though because it gives ListenTCP a chance to keep up with the source. It would be most efficient to avoid splitting to the individual flow files if possible. Since you are merging things together before HDFS, it shouldn't matter if you are merging many flow files with one message each, or a few flow files containing thousands of messages each. It just comes down to whether you want to rewrite some of the logic in your custom processor. ... View more

Alright I don't see anything wrong jumping out at me, this is what has worked for me: https://community.hortonworks.com/questions/63180/error-in-nifi-flow.html#answer-63240 Usually when it gets as far as prompting for a password, it is because something with your keytab and principal was not correct, maybe try listing the keytab and making sure the principal you are using is actually in that keytab. ... View more

The single biggest performance improvement for ListenTCP will be increasing the "Max Batch Size" from 1 to something like 1000, or maybe even more. The reason is because it will drastically reduce the number of flow files produced by ListenTCP, which will drastically reduce the amount of I/O to the internal NiFi repositories. The downside is you won't have a single message per flow file anymore so your validation needs to work differently. If you can change your custom processor to stream in a flow file, read each line, and only write out the validated lines to the output stream then it should work well. If the validation processor is still the bottleneck, you could increase the concurrent tasks of this processor slightly so that it keeps up with the batches coming out ListenTCP. ... View more

It takes a comma separated list of files so you can specify: /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml Obviously using the appropriate paths on your file system. ... View more

For Java you need to build a custom processor and package it in a NAR. There is a Maven archetype to generate the proper project structure: https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions The JARs you need would be dependencies of your processors project listed in the pom.xml Then you deploy the NAR file to the lib directory of NiFi and restart. ... View more