Yes the message key is the same thing as record key. Ok so for your use case, you need more than just the same partition for all messages from a flow file. You want the same partition for all flow files from a given source. In that case, we should have a property in the processor for the partition that supports expression language, so then you could do something like... GetFile (source1) -> UpdateAttribute (set kafka.partition = 1) -> PublishKafka (partition = ${kafka.partition}) GetFile (source2) -> UpdateAttribute (set kafka.partition = 2) -> PublishKafka (partition = ${kafka.partition}) I'll clarify this on the JIRA. In the meantime you probably have a better chance of using PublishKafka_0_10 (the non-record version)... If you strip off the header before reaching this processor, then set the Message Demarcator for PublishKafka_0_10 to be a new-line, and set the key to ${filename}, you should get what you are looking for. ... View more

@Alvin Jin Also, I know you already implemented a custom service, but there is also some work here by one of the Apache NiFi committers: https://github.com/apache/nifi/pull/1938 ... View more

Site-To-Site does not do anything to the contents of your flow files, if you have 3 flow files then it transfers 3 flow files. That statement is saying that site-to-site is optimized for a continuous flow of large amounts of data, so if you run a test with 3 flow files, it probably will send all 3 flow files to only of the nodes in your cluster because it wasn't enough data to reach the point where it would start sending to the other nodes. ... View more

1) Doesn't really matter, it just needs to be shared location that all nodes can access. 2) You can do this and it might work well for small files and small amounts of files, but typically the whole point is to perform the "fetch" in parallel, where as here all the files have to be fetched on primary node (GetFile) and then all of their contents have to be redistributed to the cluster, instead of just the listings. ... View more

Generally you only want Primary Node only for a source processor like ListHDFS where you only want to perform the listing one time. ... View more

No, you have a 3 node cluster, lets say node #1 is primary node... MiNiFi is sending data to all nodes so the data is already divided across all the nodes, but you are only scheduled to process it on node #1, so now data on nodes #2 and #3 will just sit there and never get processed. ... View more