The way to set this up in Apache NiFi 1.0.0 is to use a single NiFi cluster with process groups for each team, and restrict the permissions appropriately so that members of a team can only work with in their given process group. See the "Multi-Tenancy" section of this post for an example: http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy ... View more

There could be many other things wrong, but in your ZooKeeper properties, the servers should be the same on every node. If instance #1 is pointing to 2888:3888 for all three ZK servers, then instance #2 and #3 should also be pointing to the same thing. ... View more

Yup, if you specify the Message Demarcator it will write all of the messages received in a single poll from Kafka to a single FlowFile, so it would be a maximum of "Max Poll Records" per flow file, but could be less if it received less from Kafka. ... View more

I think there are two different things, one is understanding the performance, and another is validating the data/correctness. For performance, you should be able to get a good idea of the performance by looking at the various statistics in NiFI, there are stats on each processor, process groups, and from the global menu Summary page, they all show things like FlowFiles in/out and bytes in/out. For validating the data, I don't know there is a great way to do this other than checking what ends up in HDFS. Most of the time a dataflow is a continuous stream of data and not really a discrete set where you know how many records. One last thing to mention, do you really want to write 1 million small files to HDFS? You would probably get a lot better performance by using the batching capability on ConsumeKafka to write a couple of hundred, or even thousand, messages to a single flow file. ... View more