Triggering NiFi from a Scheduler

[#1]

Hi,

We are planning to trigger NiFi using a scheduler(Control-M). I would like to know if incase of any failure, does NiFi return any return code(failure code/message), so that the scheduler can catch it & create an alert(ticket) ?

Thanks

[#2]

Hi @Rohit Ravishankar,

Could you elaborate more on what you are trying to achieve with NiFi? NiFi doesn't really have a concept of a 'job' or a 'batch' which you would trigger. The usual workflow with NiFi is for it to wait for data to appear and then process that data. Waiting for data could be listening for syslog messages, or waiting for files to appear in a directory or many other options.

Processing data is done using a flow where each processor performs a different step. Each processor has a concept of input and outputs (usually success and failure but not necessarily - it depends on the processor). Even if files are routed to a 'failure' relationship it doesn't necessarily mean the job has failed. For example, lets presume there is a file of JSON records I want to convert to AVRO records; if only one record fails conversion just that record will be routed to failure. This could then be routed to another converter with a different (maybe more general schema).

Does this helps!

[#3]

Hi @Sebastian Carroll

Thanks for your reply!

I understand NiFi doesn't really have a concept of a 'job' or a 'batch' & its like stream processing. I am using the scheduler to start a NiFi process group. If incase any proessor fails due to some reason and my flow gets hanged(does not proceed further & even records/file do not move to the failure relationship), is there a failure code which NiFi return, which we can capture & alert the users?

[#4]

Hey @Rohit Ravishankar, could you move this into a comment on my answer? Otherwise it will be difficult to follow that it is a reply to what I posted above

[#5]

Hey @Rohit Ravishankar, thanks for moving this to a comment!

Short answer: Not Really

Longer answer: It's not really what NiFi does

It still sounds like you want NiFi to process a batch. Something like users put files into a directory and then after some external trigger start the job and report any failures. Is there a reason why NiFi cannot just continually listen for input? Does it have to sync up with other 'jobs' further up or down stream? If so, a better use would be to have the scheduler (e.g. control m) 'trigger' NiFi just by moving the files into the pickup directory for example.

Apologies, I can't be more specific without more details on your use case