@Mohamed Hossam I think you are missing space in search value property. Use the below regex in search value property ^(.*?) (.*?) IP (.*?) > (.*?) .*$ (or) ([^\s]+)\s([^\s]+)\sIP\s(.*)\s>\s([^\s]+).* Use any of the above regex's. Config:- ... View more

@Lanic - With release of Apache NiFi 1.7 and HDF 3.2 in mid 2018, the ability in terminate threads still executing on a processor that is in a state of "stopping" is now possible. After changing state of processor from start to stop, you will see processor display red square. You should give the running threads an opportunity to complete their execution. If it appears the processor is just not going to stop (hung threads) you can right click on the processor and select "Terminate" from the context menu displayed as follows: ... View more

@Jose Paul - A bin would be eligible for merge with only 1 FlowFile in it since you set minEntries to 1. - When the Processor get scheduled to execute (based on configured run schedule and scheduling strategy), It will look at one of possible many incoming connections and look at only the queued FlowFile at that exact moment in time. It will then bin those FlowFiles based on configuration. So it multiple FlowFiles happen to exist in that connection with sam filename attribute value, they will be placed in same bin. At completion of of placing those FlowFiles in bins, the bins are evaluated if they are eligible to be merged. In your case since minEntries is 1 all bins with 1 or more FlowFiles would be merged. - If you run schedule is set to run as fast as possible (Timer Driven with run schedule of 0 sec), it may be reading the inbound connection so fast that it only contains 1 or just a few FlowFiles per execution. - The other scenario is an inbound connection with over 500 queued FlowFiles at time of execution. If we assume there are more than 500 FlowFiles with unique values assigned to the filename attribute, each would end up be placed in new bin (correlation attribute config). As soon as bin 500 has a FlowFile assigned to it and MergeContent tries to bin unique filename number 501, it has no available bins left so it forces the merging of the oldest bin to free a bin. - Thank you, Matt ... View more

@Andy Gisbo Yes, that guide is accurate example of using OpenID with google. ... View more

@Jose Gonzalez You can specify more then one host but it is not required. Once the RPG establish a connection to the target host it will retrieve the S2S details of the target cluster and store that locally. If the host you provided become unavailable at anytime after that initial connection , it will try anyone of the other nodes it learned about previously to get S2S details. Having multiple nodes configured helps when NiFi by giving the source Nifi more then one target node to establish initial connection with. - Your load-balancing issue is completely unrelated to how many nodes URLS you configured in your RPG. Here is an article that covers how load-balancing works with an RPG: https://community.hortonworks.com/content/kbentry/109629/how-to-achieve-better-load-balancing-using-nifis-s.html - Thanks, Matt ... View more

@Matt Clarke Thanks for the response, yes I set the max timer driven event count to 40. I might have to increase it. I will set the number of flow files to 15000 to 20000 and I shold also set the max age, so that no flow files are lingering on, will comeback with results. Thanks Dheeru ... View more

@dhieru singh Min number of entries must be set and defaults to 1. That is fine as long as you don't set max num entries. You are correct it is (min number of entries AND min group size) OR max number of entries OR max group size. So either of the "max " settings will force a merge just like max bin age will. ... View more

Thx for answer. We are using a older version today. As showed only some of the fields is supported for Expression Language. HDF Version 2.1.1 - Powered by Apache NiFi - Version 1.1.0.2.1.2.0-10 ... View more

@Matt Clarke Thanks for the reply, appreciate it. In my case directory from which files will be listed exist only on one node. For now I am trying to implement with ListSFTP and FetchSFTP processors and hope it works fine. Thank you once again for your valuable suggestions. Thanks, Basant ... View more

@Shu @dhieru singh By default their is no guaranteed order in which FlowFiles are pulled from he queue feeding any given processor. This is because NiFi favor performance over order. If you want enforce some sort of order in which FlowFiles are pulled from a inbound queue, you must add a "Prioritizer" to the inbound connection. By default, no prioritizers are added. To apply a prioritizer, simply drag the desired prioritizer(s) to the "Selected Prioritizers" box. Regardless of strategy used in your DistributeLoad processor (round Robin or next available), There will not be a continuos order to the FlowFiles queued to either MergeContent processor. Thanks, Matt ... View more