I am currently using scroll elastic processor in nifi to read data from the previous day based on timestamp. I want to schedule this activity once a day. The problem I am facing with schedule is that when I set a cron interval, the scroll elastic processor fetched the data only once i.e. 10000 records which is the limit and does not continue to fetch more records based on the input query.
Increase the Back Pressure Object Threshold and Back Pressure Data Size Threshold(if needed) after scroll elastic processor to some big number.
1.Right Click on Success Queue
2.Click on Configure
3.Goto Settings Tab
Change Back Pressure Object Threshold and Back Pressure Data Size Threshold(if needed)
Then the processor is allowed to run until Back Pressure Object Threshold (or) Back Pressure Data Size Threshold reaches max configured size
Thanks for the response. I tried increasing the back pressure of the success queue. However, I still get just one flowfile of 10000 records. It seems to be so because scrollelastic processor reads the elastic index and get data in batches of 10000 records. So it just reads it once when scheduled by cron.