nifi processor cron schedule-avoiding multiple run

[#1]

Hi,

I have scheduled a getfile processor in nifi on cron schedule, running every minute (0 0/1 * * * ?).

The above runs properly and gets a file every minute, but the processor is running multiple times(atleast 2-3) and showing warning almost every time (because it could not ingest the file, as it has ingested already and no file is present).

How to avoid this, and make the processor stop till next schedule once the file is ingested.

Could you please suggest

[#2 · Accepted solution]

@Hadoop User

1. Use ListFile Processor and run that in cron schedule for every minute, this processor will store the state and wont return any warning if there is no new file.
2. Then you can use FetchFile processor to pull the listed files from ListFile processor.
3. As these processors won't delete the file from your directory once fetch has been done(like getfile processor),
4. if you want to delete those files from directory then use ExecuteStreamCommand processor and write a shell script which can get the filename from to flowfile attribute and pass that attribute to your script.

Flow:-

1.ListFile //list all the files from directory.
2.FetchFile //fetch the listed file.
3.ExecuteStreamCommand //shell script to delete file from directory.

Refer to below link how to pass attributes to the ExecuteStreamCommand processor script.

https://pierrevillard.com/2016/03/09/transform-data-with-apache-nifi/