@J. D. Bacolod Have you considered using the PutDistributedMapCache and GetDistributedMapCache processors? Have two separate dataflows. One runs on a cron and is responsible for obtaining the token and write that token to the distirbutedMapCache using the putDistirbutedMapCache processor. The Second flow is for doing all your other operations using that token. Just before the invokeHTTP processor add a GetDistibutedMapCache processor that reads the token from the distributed map cache in to a FlowFile attribute. You then use that attribute to pass the token in your connections. One thing to keep in mind is that it is possible that a new token may be retrieved after a FlowFile had already retrieved the old token from the distirbutedMapCache. This would result in auth failure. So you will want your flow to loop back to GetDistributedMapChace processor to get latest key on auth failure on your invokeHTTP processor. This flow does not keep track in any way when a token expires, but if you know how long a token is good for you can set your cron accordingly. Thanks, Matt ... View more

@Anthony Murphy Is this a NiFi cluster or standalone NIFi instance? Make sure that the following property is set to "true" in the nifi.properties file on every instance of NiFi: nifi.flowcontroller.autoResumeState=true If this property is set to false it will trigger all components to come up stopped when Nifi is restarted. Thanks, Matt ... View more

@Alvin Jin My suggestion would be use something like: https://regex101.com/ You can enter your regex and sample test you want to run it against. Matt ... View more

@Kiran Hebbar Hello, Your question is not very clear as to what you are looking for. I am going to assume you are asking how to view the metadata currently associated with a FlowFile passing through your NiFi dataflow(s). There are several ways to view this metadata: Right click on a connection that has queued data and select "List queue" from the context menu. Form the new UI that opens you will se a list of FlowFiles. Click on the icon to the left of any one of the FlowFiles to "view details" of that FlowFile. There you will find a "attributes" tab that list all the key/value pairs associated to this FlowFile. Use data provenance to perform a search on FlowFile events. "Data Provenance" can be found under the upper right corner hamburger menu in the NiFi UI. Click the search icon to open a "Search Events" UI where you can add criteria to limit the results (Provenance returns 1000 of the most recent events). From the final list use the same "view details" icon to the left of an event to open a new UI that will show the Attributes of your selected FlowFile. Use the LogAttribute processor. Add this processor anywhere in your dataflow. As FlowFile pass through this processor their FlowFile Attributes as they exist at the time of passing through this processor will be logged to the nifi-app.log. Keep in mind that this processor can greatly increase the size of your logs and require more space to store your logs. If you found this answer addressed your question, please mark it as accepted. Thanks, Matt ... View more

@Alvaro Dominguez The primary node could change at anytime. You could use postHTTP and listenHTTP processor to route FlowFiles from multiple nodes to a single node. My concern would be heap usage to merge (zip) 160K FlowFiles on a single NiFi node. The FlowFile metadata for all those FlowFiles being zipped would be help in heap memory until the zip is complete. Any objection to having a zip of zips? In other words you could still create 4 unique zip files (1 per node each with unique filename), then send these zipped files to one node to be zipped once more in to a new zip with the single name you want written into HDFS. Thanks, Matt ... View more

@Alvaro Dominguez The only way to create one single zip file is to have one node perform the zipping of all the files. This sounds less then ideal. How large are each of these individual zip files and how many FlowFiles on average go in to each zip file? ... View more