Support Questions
Find answers, ask questions, and share your expertise

scheduling nifi procesor to run more than once

Explorer

I am currently using scroll elastic processor in nifi to read data from the previous day based on timestamp. I want to schedule this activity once a day. The problem I am facing with schedule is that when I set a cron interval, the scroll elastic processor fetched the data only once i.e. 10000 records which is the limit and does not continue to fetch more records based on the input query.

3 REPLIES 3

Super Guru

@Rohan Naidu

Increase the Back Pressure Object Threshold and Back Pressure Data Size Threshold(if needed) after scroll elastic processor to some big number.

Steps:-

1.Right Click on Success Queue

2.Click on Configure

60437-queue-configure.png

3.Goto Settings Tab

Change Back Pressure Object Threshold and Back Pressure Data Size Threshold(if needed)

60436-queue.png

Then the processor is allowed to run until Back Pressure Object Threshold (or) Back Pressure Data Size Threshold reaches max configured size

Explorer

Hi @Shu,

Thanks for the response. I tried increasing the back pressure of the success queue. However, I still get just one flowfile of 10000 records. It seems to be so because scrollelastic processor reads the elastic index and get data in batches of 10000 records. So it just reads it once when scheduled by cron.

Explorer

Still looking for some help on this. Need scrollelastic to run once a day for more than 10,000 records.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.