nifi processor cron schedule-avoiding multiple run

mark_hadoop — Mon, 30 Oct 2017 23:07:19 GMT

Hi,

I have scheduled a getfile processor in nifi on cron schedule, running every minute (0 0/1 * * * ?).

The above runs properly and gets a file every minute, but the processor is running multiple times(atleast 2-3) and showing warning almost every time (because it could not ingest the file, as it has ingested already and no file is present).

How to avoid this, and make the processor stop till next schedule once the file is ingested.

Could you please suggest

Re: nifi processor cron schedule-avoiding multiple run

Shu_ashu — Tue, 31 Oct 2017 00:00:53 GMT

@Hadoop User

Use ListFile Processor and run that in cron schedule for every minute, this processor will store the state and wont return any warning if there is no new file.
Then you can use FetchFile processor to pull the listed files from ListFile processor.
As these processors won't delete the file from your directory once fetch has been done(like getfile processor),
if you want to delete those files from directory then use ExecuteStreamCommand processor and write a shell script which can get the filename from to flowfile attribute and pass that attribute to your script.

Flow:-

1.ListFile //list all the files from directory.
2.FetchFile //fetch the listed file.
3.ExecuteStreamCommand //shell script to delete file from directory.

Refer to below link how to pass attributes to the ExecuteStreamCommand processor script.

https://pierrevillard.com/2016/03/09/transform-data-with-apache-nifi/

question nifi processor cron schedule-avoiding multiple run in Support Questions

nifi processor cron schedule-avoiding multiple run

Re: nifi processor cron schedule-avoiding multiple run