@Abhinav Joshi What was the reason given for why the UpdateAttribute processors were now invalid? ... View more

@Isha Tiwari - The "Max" configuration properties do not govern how long a bin waits to be merged. The Merge based processors work as follows: - 1. The processor executes based upon the configured run Schedule. 2. At exact time of execution the Merge processor looks at what FlowFiles exist on Inbound connection that have not already been allocated to a Merge processor bin. 3. Those FlowFiles are then allocated to one or more bins. The max Bin size and Max number records create a ceiling for how many FlowFiles can be allocate to a bin. If a bin has reached one of these max values, additional FlowFile in this current execution start getting allocated to a new bin. 4. Once all FlowFiles from this current execution (Thread does not keep checking for new FlowFiles coming in to inbound connection. Those new FlowFiles would be handled by next execution) have been allocated to one or more bins, those bins are evaluated to see if they are eligible to be merged. In order to be eligible the bin must meet both minimum settings for size and number of records or the max bin age has been reached. In your case, a bin could be binned with only 20 records and 20 KB of size or if a bin has existed for at least 1 minute. - If you find your merging small bins consistently, changing the run schedule on your merge processor should help. This would allow more time between executions for FlowFiles to queue on the inbound connection. - IMPORTANT: Keep in mind that all FlowFiles allocated to bins are bing held in heap memory (swapping does not occur with bins). Specifically the FlowFile attributes/Metadata is the portion of the FlowFile held in heap memory. Your max records of 100,000 could result in considerable heap pressure. Using two Merge processors in series could achieve same result with lower heap usage. - I use MergeContent in following Article about connection queues as an example: https://community.hortonworks.com/articles/184990/dissecting-the-nifi-connection-heap-usage-and-perf.html - Thank you, Matt -If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@Mario Tigua The File Filter property in the listFile processor does not support NiFi Expression Language. If yo float your cursor of the question mark icon to the right of a property name it will display a pop-up window that will tell you if this property will support NiFi expression language. - This property expects a java regular expression instead. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@TRACEY JACKSON NiFi processors are configurable to run using a Timer Driven or Cron Driven scheduling strategy. This scheduling can uniquely configured per each processors configuration. Processors are then started and the operate based on the configured scheduler from that point forward. - The NiFi API can be used to perform any action a user can perform directly from the UI. The best way to learn the rest-api calls is to use the "Developer tools" available via most browsers. - While focused on tab where NiFi UI is open, launch the developers tools. Chrome Browser example: Then perform the action via the NiFi UI and you will see the "Network" call display in list. You can right click on that call and select "Copy as curl". Now you have an example of how to execute that same request via command line. - For more info in the NIFi rest-api, you can look in "help" found within the NiFi UI Global menu (upper right corner). The Rest Api documentation can be found in the "Developer" section at bottom of list on far left side of help UI. - The danger with trying to "run" dataflows in NiFi via command line is you may end up stopping processors in the dataflow chain that results in FlowFile being unprocessed and sitting on connection queues between processors. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more

@sri chaturvedi - The UpdateAttribute processor state capability can only store state locally on each node in the cluster. Other nodes in your cluster have no idea what local state value has been stored on other nodes in your cluster. - So I suspect one or both of the following is occurring: 1. The upstream dataflow data originates on the Primary node only (For example source ingest processor runs "primary node" execution). On NiFi restart a different node in your 3 node cluster is elected as primary node and now the FlowFiles traversing this dataflow are on a different node where there is no previous state, so it appears as if state started over. This would explain the appearance of a reset on cluster restart. 2. This UpdateAttribute processor is receiving inbound FlowFiles on all three nodes. Since each node stores its own state locally for this processor, each would be incrementing its own count independently of each other. This would explain the -duplicate state values seen on some FlowFiles. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more