When you are running in a cluster, there should be a clustered state manager defined in conf/state-management.xml which likely be the ZooKeeper cluster state manager. If you are using a correctly setup ZooKeeper quorum, typically 3 nodes, then you really shouldn't need to back up anything since the information would be replicated on all 3 ZK instances. ... View more

Is there any pattern you can find to reproduce this? If you restart NiFi Registry after this happens, is it working ok again, or does it have an issue on start up? ... View more

Thanks for the insight, that makes total sense. Now that you mentioned provenance, I'm now wondering if NiFi should automatically be putting the bundle info of the component into the provenance events, since that is important information to know when looking at the history. ... View more

If there is a good reason for a processor to know the bundle information, then I think we can provide it through one of the context APIs. I'm curious to know what the use case is for the processor needing to know this, not saying we shouldn't add it, just want to understand. ... View more

The processor has a pool of consumers that is created when the processor is started so you would have to use the REST API to stop the processor, change the value of the topic names property, and then start the processor again. ... View more

FetchHDFS is to fetch a file, and PutHDFS is to write a file... so if there is a directory with no files then there is nothing to fetch or put. ... View more

Your whole flow will perform better if you have flow files with many records, rather than 1 record per flow file. PutDatabaseRecord will set auto-commit to false on the connection, then start executing the statements, and if any failure happens it will call rollback on the connection. ... View more

When you create a topic there are two different concepts - partitions and replication. If you have 3 brokers and create a topic with 1 partition, then the entire topic exists only on one of those brokers. If you create a topic with 3 paritions then 1/3 of the topic is on broker 1 as partition 1, 1/3 on broker 2 as partition 2, and 1/3 on broker 3 as partition 3. If you create a topic with 3 partitions AND replicaiton factor of 2, then its same as above except there is also a copy of each partition on another node. So parition 1 may be on broker 1 with a copy on broker 2, parition 2 maybe be on broker 2 with a copy on broker 3, and partition 3 may be on broker 3 with a copy on broker 1. In general, replication ensures that if a broker goes down then another broker still has the data, and partition allows for higher read/write throughput by dividing up the data across multiple nodes. ... View more

It only supports expression language from system properties and environment variables, but not from incoming flow files. This is because when the processor is started it will initialize a pool of UDP clients, so the port and host must be known before hand. It would probably be possible to make the UDP case more dynamic, but part of this issue is that PutUDP and PutTCP share a significant amount of underlying code. ... View more

All of the NARs are unpacked under NIFI_HOME/work/nar so even though you deleted the NAR, it was still present in the work directory. It should be automatically determining that the NAR changed and unpacking it again, but if it is not then you can always blow away the work directory, or blow away the specific directory for your NAR under the work directory. ... View more

I've responded on stackoverflow... https://stackoverflow.com/questions/51835130/how-exactly-apache-nifi-consumekafka-1-0-processor-works ... View more

https://pierrevillard.com/2017/01/24/integration-of-nifi-with-ldap/ https://ijokarumawak.github.io/nifi/2016/11/15/nifi-auth/ https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider ... View more

If ListFTP is showing an error in the UI then that error message is in nifi-app.log somewhere, please provide the full stacktrace that goes with that error. ... View more

Custom components have access to an API, not to the implementations. So a processor has access to ProcessSession which is the API, but not StandardProcessSession which is the implementation. This is done on purpose so that the NiFi framework can evolve behind the scens as long as it adheres to the API which is the contract with the processors/components. The only way to get the information you are looking for would be to have an upstream processor add attributes to the flow file containing the relevant information, and then use session.getAttribute(...) to get the attributes you are interested in. ... View more

When you say it is "not working", what exactly is happening? There can only really be three possible outcomes... a) it fetched successfully b) it did not fetch successfully and there is a red error on the processor in the UI, and an error in nifi-app.log c) it is stuck connecting and there is a little number icon in the top-right of the processor which shows that 1 thread is still running trying to execute Which of these choices describes the result? ... View more