Helping setting up cron-based nifi processor

[#1]

I'm trying to setup a nifi processor to run once daily, using the 'cron' option under scheduling.

Under "run schedule", I put "01 18 * * * ?" , which should be 6:01pm (I couldn't get nifi to accept it without the question mark).

But this causes the scheduler to run on the hour, every hour.

Can someone please help me with my syntax here?

[#2 · Accepted solution]

@Zack Riesland

The Cron you have there should run 18 minutes and 1 second into every hour.

What you really are looking for as a cron here for 6:01pm is ( 0 1 18 * * ? * or 0 1 18 * * ? )

• CRON driven: When using the CRON driven scheduling mode, the Processor is scheduled to run periodically, similar to the Timer driven scheduling mode. However, the CRON driven mode provides significantly more flexibility at the expense of increasing the complexity of the configuration. This value is made up of seven fields (where the seventh field is optional), each separated by a space. These fields include:
  • Seconds
  • Minutes
  • Hours
  • Day of Month
  • Month
  • Day of Week
  • Year

The value for each of these fields should be a number, range, or increment. Range here refers to a syntax of <number>-<number>. For example,the Seconds field could be set to 0-30, meaning that the Processor should only be scheduled if the time is 0 to 30 seconds after the minute. Additionally, a value of * indicates that all values are valid for this field. Multiple values can also be entered using a , as a separator: 0,5,10,15,30. An increment is written as <start value>/<increment>. For example, settings a value of 0/10 for the seconds fields means that valid values are 0, 10, 20, 30, 40, and 50. However, if we change this to 5/10, valid values become 5, 15, 25, 35, 45, and 55.

For the Month field, valid values are 1 (January) through 12 (December).

For the Day of Week field, valid values are 1 (Sunday) through 7 (Saturday). Additionally, a value of L may be appended to one of these values to indicate the last occurrence of this day in the month. For example, 1L can be used to indicate the last Monday of the month.

Thanks,

Matt

[#3]

NiFi does not use a linux cron. It uses a Quartz cron/scheduler.

[#4]

When I enter "0 0 18 * * * ?", I get this error message: "Scheduling Period '0 0 18 * * * ?' is not a valid cron expression: '?' can only be specified for Day-of-Month or Day-of-Week"

However, "0 0 18 * * ? *" seems to work.

What is the difference between the meaning of "*" and "?" ?

[#5]

@Zack Riesland

There are seven fields; however, the seventh field is optional. So you are correct.

so both " 0 0 18 * * ? " and " 0 0 18 * * ? * " are valid.

The below is from http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/crontrigger.html

----------------

• * (“all values”) - used to select all values within a field. For example, “” in the minute field means *“every minute”.
• ? (“no specific value”) - useful when you need to specify something in one of the two fields in which the character is allowed, but not the other. For example, if I want my trigger to fire on a particular day of the month (say, the 10th), but don’t care what day of the week that happens to be, I would put “10” in the day-of-month field, and “?” in the day-of-week field. See the examples below for clarification.

-----------------

so only fields 4 and 6 will accept ?.

Thanks,

Matt

[#6]

Hi @zack_riesland 

can you provide corn expression for this timing 12:00PM and 11:40AM 

[#7]

@hargav  
It is not possible to create single Quartz cron that will schedule at 12:00:00 PM and 11:40:00 AM each day.

However, the following would execute at 11:40:00AM and 12:40:00PM every day

0 40 11,12 * * ?

 
and following would execute at 11:00:00AM, 11:40:00AM, 12:00:00PM, and 12:40:00PM

0 0,40 11,12 * * ?

 or

0 0/40 11,12 * * ?

Hope this helps you,

Matt

[#8]

Hi @MattWho 

I want corn driven expression for 11:40 am separate and I want corn driven expression for 12:00pm separate

Please send those expressions

[#9]

@hargav 

NiFi processor scheduled every day at 11:40AM:

0 40 11 * * ? *

NiFi processor Scheduled every day at 12:00PM:

0 0 12 * * ? *

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#10]

Hi @MattWho  

I was used below corn expression for merge record but the data  was not went to the queue in nifi platform ,We need to change any configuration in merge record ?

please suggest me any solution 

0 40 11 * * ? *

[#11]

@hargav 
Please create a new community question for your queries around MergeRecord processor.  This is the best way to get attention and best for community to have a separate thread for each specific query.  

I am not clear on your use case for using "cron driven" scheduling with the MergeRecord.  This would not be a common thing to do. Best to explain your use case in a new community thread along with sharing your MergeRecord processor configuration.  Feel free to @MattWho in the new community post to notify me.

Thanks,

Matt

[#12]

Thanks Matt,

So so would "0 0 18 * * ?" indicate that the job should be run 18 hours into every day?

[#13]

The Quartz scheduler has 7 fields, so the cron would need to be 0 0 18 * * ? *. The seventh field is optional for year. Yes the cron you have there will run the 18th hour of every day.

[#14]

Hey @Matt,

I am using a cron job scheduling strategy too and my cron job is "0 30 13 * * ?" similar to what is discussed here. But, for some reason, it just runs for the day I create it and doesn't repeat the next day. Do you know the reason why? I keep NiFi running and don't think it's shutting down for the cron job to not execute. Any help appreciated.

[#15]

@raghav130593 

Where "*" indicates every value, the "?" can mean i don't care about the value.

So you want to have corn execute at 17:30:00 every day, So i would recommend a "*" instead of "?" in the day of week field.

A good example of when to use a ? in a supporting field would be this cron:
0 30 17 5 * ?
The above says execute at 17:30:00 on the 5th of every month.  The day of week field here is a "?" mark because the 5th of every month could fall on any one of the days of the week (will vary from month to month).

Thanks,

Matt

[#16]

@zack_riesland 
Yes, "0 0 18 * * ?" means schedule to run at 0 secs, 0 mins, 18 hour, every day, every month, any day of week"

What's important to understand is the quartz cron is used to schedule the processor to execute.  In order for the processor to execute at exactly 18:00:00, NiFi must have an available thread in the NiFi timer driven thread pool in order to execute.  If a thread is not available, the processors code will execute as soon as a thread become available.  Since it has been "scheduled" it will run as soon as thread becomes available and then will get scheduled again the next day at 18:00:00.

Matt

[#17]

Below is the syntax I have tried with 0 30 0 * * ? or 0 30 1 * * ?, which should run at 12:30 or 1:30 midnight respectively.

Also I have tired running the flow with cron in nifi for every 20 minutes duration, it worked perfectly.

But we want it to be run, daily once at particular time as mentioned above example (0 30 0 * * ? or 0 30 1 * * ?).

But it is not working. can anyone please guide.

Thanks.

[#18]

Hi @has_sha786  

even i was facing same issue ,i was setted to run my processor at 12:00pm(0 0 12 * * ?) and another one is 11:45am(0 45 11 * * ?) but it's not working 

if u have any suggestions please reply me 

[#19]

Superb. Thanks for this great information. the 7 fields is something that I wasn't aware of . drilling down at a second level is great ! @Matt Clarke great answer