@TRACEY JACKSON NiFi processors are configurable to run using a Timer Driven or Cron Driven scheduling strategy. This scheduling can uniquely configured per each processors configuration. Processors are then started and the operate based on the configured scheduler from that point forward. - The NiFi API can be used to perform any action a user can perform directly from the UI. The best way to learn the rest-api calls is to use the "Developer tools" available via most browsers. - While focused on tab where NiFi UI is open, launch the developers tools. Chrome Browser example: Then perform the action via the NiFi UI and you will see the "Network" call display in list. You can right click on that call and select "Copy as curl". Now you have an example of how to execute that same request via command line. - For more info in the NIFi rest-api, you can look in "help" found within the NiFi UI Global menu (upper right corner). The Rest Api documentation can be found in the "Developer" section at bottom of list on far left side of help UI. - The danger with trying to "run" dataflows in NiFi via command line is you may end up stopping processors in the dataflow chain that results in FlowFile being unprocessed and sitting on connection queues between processors. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@sri chaturvedi - The UpdateAttribute processor state capability can only store state locally on each node in the cluster. Other nodes in your cluster have no idea what local state value has been stored on other nodes in your cluster. - So I suspect one or both of the following is occurring: 1. The upstream dataflow data originates on the Primary node only (For example source ingest processor runs "primary node" execution). On NiFi restart a different node in your 3 node cluster is elected as primary node and now the FlowFiles traversing this dataflow are on a different node where there is no previous state, so it appears as if state started over. This would explain the appearance of a reset on cluster restart. 2. This UpdateAttribute processor is receiving inbound FlowFiles on all three nodes. Since each node stores its own state locally for this processor, each would be incrementing its own count independently of each other. This would explain the -duplicate state values seen on some FlowFiles. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Sean Dockery 1. The problem with G1GC in Java 8 is not unique to provenance. G1GC in Java 8 and earlier was still considered experimental. It was observed that G1GC had better performance with the larger heap sizes commonly used in NiFi setups, so early on it was the recommended GC for NiFI. While these bugs exist that can cause corruption in the in the Java heap space, we had not encountered the corruption prior to the introduction of the new WriteAheadProvenance implementation. While the G1GC issues have been resolved as of Java 9, those fixes were not back ported to earlier versions of Java. NiFi currently only supports Java 8, so we decided to move away from recommending using G1GC when using the new high performant WriteAheadProvenance implementation. https://wiki.apache.org/lucene-java/JavaBugs 2. The change of default provenance implementation from PersistentProvenance to WriteAheadProvenance only occurred recently and it appears no one changed the configuration in the bootstrap.conf at that time to comment out the G1GC line. It is also very likely we will again recommend G1GC once NiFi supports newer versions of Java where these G1GC issues have been addressed. 3. I have no answer for number 3, perhaps someone else can comment on that question in your query. Thank you, Matt If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Adi Sutanto - Provenance replay inserts the FlowFile back on to the inbound connection to the processor. Since the GetFile processor does not have any inbound connections, there is no where to insert a FlowFile for replay. - As far as PutSFTP processor, you are correct that it generates "SEND" and "DROP" events. If you are auto-terminating the "failure" and/or "reject" relationships, you should see "DROP" events in provenance from which you can perform a "Replay". If these relationships are being routed via 1 or more connections, then your FlowFile should be sitting in one of those connections allowing you to redirect those FlowFile back to the PutSFTP to try again. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Daniel Nguyen I see a mistake in your authorizers.xml: <property name="CN=localhost, OU=NIFI"></property> The above line should actually be: <property name="Node Identity 1">CN=localhost, OU=NIFI</property> Keep in mind that NiFi only builds the users.xml and authorizations.xml files from the authorizers.xml file configuration if those files do not already exist. So once you fix your authorizers,xml, you will need to delete or rename your existing users.xml and authorizations.xml files before starting NiFi. - Thanks, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Bala S I would recommend using capture groups in your regex. Here is an example that will work for your example: I then replace with just the value from capture group 1 - Here is another example that is more generic (can have many ";<somevalue>" after first one: Here again I use capture group 1 as my replacement value. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link ... View more