Hi, We are currently running NiFi as a single instance & we are planning to move to a clustered setup(3 node cluster). Please consider the below sample flow, ListFile -> updateAtrribute -> routeOnAttribute -> ExecuteStreamCommand(Executes a shell script) -> fetchFile -> updateAtrribute -> fetchFile -> putFile Since we are going to run in cluster setup, we need to use Remote Processor groups to balance the load. We going to place the RPG after ListFile processor, ListFile(on Primary Node) -> RPG Input port -> updateAtrribute -> routeOnAttribute -> ExecuteStreamCommand(Executes a shell script) -> fetchFile -> updateAtrribute -> fetchFile -> putFile My question is, if I want my ExecuteStreamCommand(which triggers a shell script) to execute only on the primary node & rest of the processors in all the nodes, can I go ahead and change the settings of processor to run 'On Primary Node'? Will it have any impact on the flow? Thanks, R.Rohit ... View more

Hi Team, I went through the NiFi documentation to get an idea about Controller Services. But still I am not clear when to go for Controller services. Could you please help me with answering the below queries? 1. What are Controller Services & its characteristics/advantages? Why do we go for it? 2. In what scenarioes we can/should use it? Thanks & Regards, R.Rohit ... View more

Hi Team, I have a scenario where I need to read a file with older timestamp, after a file with latest timestamp has been processed by the ListFile processor. Below are the details of what I tried, I am setting the Min & Max age filter in the ListFile processor as below, Min Age – 300 sec [5 minutes] Max Age – 864000 sec [10 days] Then, I touch a file in the file system with the latest timestamp as shown below, -rw-r--r-- 1 userA userB users 0 Jul 9 00:57 a.txt the file gets picked by the ListFile processor.Then, I touch a file in the file system with older timestamp as shown below, -rw-r--r-- 1 userA userB users 0 Jul 5 00:00 b.txt However this file is not getting picked by the processor. My understaning was that files whose modified time is between 5 minutes to 10 days old, should get picked. Could you please let me know the actual behaviour of the Min/Max filter? Also could you please let know whether the scenario which I have explained above can be achieved? Thanks & Regards, R.Rohit ... View more

Hi Team, I have a set of processors running with default Back Pressure Data Size Threshold of 1 GB. Below is the sample flow, ListFile -> FetchFile -> CustomProcessor -> PutHDFS The Custom Processor's concurrency is set to 4. I encounter a single file whose data size is 31 GB. I wanted to know how the data will get processed in this case? Will back Pressure get applied ? Thanks! ... View more

Hi Team, In my file I have some records as show below, Sample Input- 1|transaction|I|2017-03-28|Current 1|master|B|2017-03-28|Current 1|master|A|2017-03-28|Current Required Output- 1|transaction|I|2017-03-28|Current 1|master|A|2017-03-28|Current While processing the file using NiFi, I want to consider only records having "I" & "A" in the third position. I am trying to use "RouteText" processor using regular expression to filter the records. However I am not successful in getting the correct expression for filtering the records. Could you please help me out? Thanks! ... View more

Team, I am getting the below files in a directory in the same order, companies.dat data.xml transaction.dat accounts.dat bankdetails.dat I use a List File processor to read the files from a directory. When we encounter ".xml" file, it should trigger a shell script & rest of the files (transaction.dat, accounts.dat & bankdetails.dat) should not be processed until script completes its execution. Once the script is complete, the rest of flow files should be processed. If .dat file then process the file(Path A). If .xml file, then trigger the script(Path B) & then process the rest of the .dat files(Path A). Could you please help us with an approach? Please let me know if more info is required. Thanks! ... View more