@Sai aditya - Let me make sure I understand what you are doing first... 1. You are using an UpdateAttribute processor to add a new attribute to each FlowFile that pass through that processor. correct? 2. This attribute on each FlowFile has an attribute name of "fnnn" and some value assigned to that attribute. correct? 3. This FlowFile then flows to an invokeHTTP processor where you need to add the value associated to that attribute to the URL. correct? Every NiFi Expression Language (EL) statement starts with "${" and then ends with "}". You will notice in above screenshot I have modified what you provided with a proper EL of "${fnnn}". - The ReplaceText processor is used to modify the content of a FlowFile. Are you looking to modify the FlowFile content? - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Micah Pearce There are numerous ways to accomplish the same result in NiFi. The use of the ${retry.counter:replaceNull('0'):plus(1)} NiFi Expression language statement is to create an attribute with value 1 if attribute "retry.counter" does not exist on the FlowFile; otherwise, the existing value currently assigned to existing attribute "retry.counter" is returned and incremented by 1. - If we just set it to one, each time the same FlowFile passes through this loop it would get set to 1 and would never increment. - Thank you, Matt ... View more

@Vivek Thakur You can put whatever value you like in there. - for example: 48 hours 30 days 90 days etc.... - Setting a value of say 90 days here does not guarantee that you will even have 1 day of provenance history. The amount of provenance history retained is going to be capped by another Provenance configuration and rate at which your dataflow(s) are producing provenance events. - nifi.provenance.repository.max.storage.size The maximum amount of data provenance information to store at a time. The default value is 1 GB. The Data Provenance capability can consume a great deal of storage space because so much data is kept. For production environments, values of 1-2 TB or more is not uncommon. - The more processors in use and the higher the number of FlowFile being processed through these processors will equate to higher number of produced provenance events. - If you are looking to store large amounts of provenance events, you may want to consider adjusting the defaults on the below setting as well to have better provenance query performance: - nifi.provenance.repository.index.shard.size The repository uses Apache Lucene to performing indexing and searching capabilities. This value indicates how large a Lucene Index should become before the Repository starts writing to a new Index. Large values for the shard size will result in more Java heap usage when searching the Provenance Repository but should provide better performance. The default value is 500 MB. However, this is due to the fact that defaults are tuned for very small environments where most users begin to use NiFi. For production environments, it is advisable to change this value to 4 to 8 GB. Once all Provenance Events in the index have been aged off from the "event files," the index will be destroyed as well. Note: this value should be smaller than (no more than half of) the nifi.provenance.repository.max.storage.size property. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link ... View more

@Aaron R The only method you can use to systematically adjust the configured run Schedule of a NiFi processor is through the use of the NiFi rest API. You would achieve this through a series of invokeHTTP processors. - 1. Use invokeHTTP processor to stop another processor. 2. Use invokeHTTP processor to change "Run schedule" configuration on a processor. 3. Use InvokeHTTP processor to start processor again. (Note that the processor will immediately execute when you start it and then not execute again for newly configured "run schedule" duration.) - You would then need to repeat above process for all three of your sensor inputs. Since I do not know the full details of your use case, it is not clear if this solution will really meet you needs. However, I can tell that based on how NiFi was developed, this is the only method for adjusting this configuration without user manual intervention. - You may just be better off building a flow that sends a FlowFile to each of the sensors invokeHTTP processor to trigger their execution. The InvokeHTTP processor does accept an inbound connection. When setup this way the invokeHTTP processor configured "run schedule" controls how often the processor checks that inbound connection for a FlowFile. When a FlowFile exists it will execute. - When a FlowFile is generated (by your "Sensor monitor controller" for example), that FlowFile will be timestamped. First you would need to split that incoming JSON into three different FlowFiles (one for each sensor). You could build a flow that parses the various sensor setting in to NiFi FlowFile Attributes and then create a if/then loop to check when the FlowFile timestamp + sensor value is equal to or less than current time. When that loop exits you route the FlowFile to appropriate invokeHTTP processor to trigger it to collect the sensor data. - Now becomes the question of what do you want to do with "Original" FlowFile out of the invokeHTTP. You can discard it and expect a new sensor monitor input each time or you can reset its trigger time and loop it back to the start. You would also need to in this case build in a method in your flow to kick this looping FlowFile out once a new sensor input comes in. (perhaps using the DistributedMapCache to store a value you increment each time a new timer comes in and then have your looping FlowFiles fetch this value to see if it has changed and kick FlowFile out if it has). - Hope these ideas help you with your design decisions. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Derek Calderon Unfortunately, a rolling restart will not work here. When you shutdown only one node another node in the cluster would get elected as cluster coordinator and still retain list of nodes known to be part of cluster. By shutting down all nodes you essential wipe the slate clean of what nodes are in cluster. On start nodes will come up, components (processors, controller services, reporting tasks. etc..) will return to last known state (running, stopped, etc..), FlowFiles will be loaded back in to last reported connection queue, and processing will continue. - One of the nodes will get elected as the cluster coordinator by Zookeeper and then other nodes will learn from ZK who was elected and start sending heartbeats directly to that elected cluster coordinator to join the cluster. As node join they will be added to list of connected nodes. - The dead node that never checks in will no longer be in the list. - No need to stop components before stopping NiFi. They will return to last known state on start. - Thank you, Matt ... View more

@Derek Calderon Since your NiFi cluster is waiting on a response from the offloading node, it will not give you an option to delete node from list thus reducing your cluster node count by one. - If you stop all your remaining nodes and then start them, he cluster will come back up with only those nodes which join during startup. The other node will no longer be present. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Benjamin Bouret The invokeHTTP processor would require you to use a SSL context service when communicating with a secure (https) endpoint. The SSLContext service can be setup with only a truststore.jks if this is only a 1-way TLS connection that does not require client authentication. - You should be able to use openssl to get the complete public certificate chain from the target secured endpoint. From command line execute following command: Openssl s_client -connect <hostname>:<port> -showcerts - The return from this command will include one or more public certificates. each public certificate will start with and end with following: -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE----- - Copy each certificate including the above two lines and write each to separate file with a .crt extension. for example: CA-1.crt - Then import each of these public certificates in to the truststore you want to use in your SSLContext service as follows: # keytool -import -alias <unique Alias name 1> -file CA-1.crt -keystore truststore.jks # keytool -import -alias <unique Alias name 2> -file CA-2.crt -keystore truststore.jks etc... - Make your your NiFi service user can read this file where ever you decide to place on each of your NiFi nodes. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Mahadevan Ganesan What NiFi processor are you using to merge your CSV files together? How do you have that NiFi processor configured? Is your NiFi a multi-node cluster or a single standalone NiFi instance? Thank you, Matt ... View more

@Rishabh Singh - Now that you have one node up and connected to cluster (meaning it is able to send heartbeats to cluster coordinator), you need to look at the nifi-app.log on your other two nodes as they are starting to see why they have not attempted to connect to cluster. - Perhaps they can't talk to zookeeper? Perhaps they can not resolve the hostname of the elected cluster coordinator for sending heartbeats? - Thanks, Matt ... View more

@Rishabh Singh The NiFi Cluster Manager (NCM) which was a dedicated manger which NiFi nodes connected to no longer exists as of the Apache NIFi 1.x releases. As @Lester Martin stated, all instances of NiFi are capable of acting as the "cluster coordinator" (replacement for the old dedicated NCM). A NiFi cluster requires Zookeeper (ZK). ZK is used to elect one node in your NiFi cluster as the "Cluster Coordinator" and one node as the "primary node". Which node is elected to these roles can change at any time. ... View more

@Ghanashyam Srinivas - What you are seeing here has has been covered in https://jira.apache.org/jira/browse/NIFI-5464 - There is no reported fix for this issue yet. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link ... View more

@praneetha satya - If the content of your FlowFile is just ascii text you can use the ReplaceText processor to edit that content. You would configured this processor with a Java regular expression that would single out "2017" in to a set of capture groups. The replacement text value would use some of those capture groups and your new value. - Here is a online regex tester: http://myregexp.com/index.html - Using a FlowFile with just "Thu, 14 Sep 2017 00:00:00 -0000", i configured a ReplaceText processor with: Search Value: (.*?, \d{1,2} \w{3}) (\d{4}) (.*)$ . <--- This creates 3 capture groups (capture group 2 is the "2017") Replacement Value: $1 2019 $3 . <--- sets a replacement value of (capture group 1)(space)(2019)(space)(capture group 3) - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Sriram The URL you just shared is http. NiFi must be secured over https before any user authentication can be enabled. You can not perform user authentication over http. - Thank you, Matt ... View more