Depends on what arguments you are providing to the hash function. If your argument values are unique, you would most likely get unique value from hash. Keep in mind hive hash function return int (which is 32bit) so you may see -ve numbers as well. You can use something like reflect('java.util.UUID','randomUUID') to generate uniqueID or comeup with some unique code. I would not suggest using hash function, if you want to generate unique ids. ... View more

Templates when created are not inherently made available to all users. They are tightly coupled to the authorization policies of the process group they are created in. Since these templates are coupled in this way to a particular process group, that process group cannot be deleted until those associated templates are removed. If you access the "templates" UI, you will see a "Process Group Id" column. You should see the process group ID for your "Templates" process group associated with two of your templates. NiFi does not give you the ability to re-associated templates with another existing process group. The idea here is to allow users who are restricted via access policies to some specific process group(s) in NiFi to upload there templates which may contain sensitive configurations and prevent other users from being able to load/use them. Of course a user can adjust the access policies once the template has been uploaded; however, the template remains associated with the Process group in which it was uploaded. Thanks, Matt ... View more

ConsumeKafka is implemented with "at-least once" guarantees, so it writes the data to a flow file and commits the session for that flow file, before committing the offsets to ensure no data loss. If an error happens committing the offsets, as can happen with the rebalance, the flow file is already committed and transferred to success and can't be undone. If it was done in the reverse order (commit offsets first, then commit the flow file session) you could route to a failure if committing the offsets failed, but you would also risk data loss if NiFi crashed after commit offsets but before committing the flow file session...this would be "at-most once" guarantees. ... View more

ConsumeKafka keeps a pool of consumers behind the scenes, equal to the number of concurrent tasks for that instance. So in a simple case with ConsumeKafka having 1 concurrent task, the first time it executes it will ask the pool for a consumer, there will be none the firs time through so it will create a new one, consume the data from Kafka and then stick it back in the pool for next time. From reading Kafka's documentation (https://kafka.apache.org/documentation) I would expect that the session timeout and heartbeat apply to the consumer while it is sitting the pool. So with the above configuration you described, I think the consumer in the pool would send heartbeats and stay active for 5 mins, then when the processor executed 5 mins later, it would have to create a new consumer from scratch. ... View more