question NiFi's GetHDFS processor with Cron schedule not reading all files in the directory in Archives of Support Questions (Read Only)

NiFi's GetHDFS processor with Cron schedule not reading all files in the directory

Raj_B — Sun, 25 Jun 2017 22:39:25 GMT

Hello,

I've NiFi (standalone instance 1.0.1) GetHDFS with this cron schedule - 0 30 0 * * ?

I want the processor to start at 12:30 AM daily; so, with the above schedule, the processor started at the expected time this morning and some files have been read, but it hasn't finished reading all the files; I had quite a few files to read in the directory yesterday and right now it still has 1200+ files left in the directory; I have the "Keep Source File" set to false, so it would/should delete the files as it reads; that shows the files left in the directory haven't been read by the processor;

My understanding is, with the above schedule, once GetHDFS starts, it should keep reading until all the files in the directory are exhausted; but I'm not understanding why some files are still left.

Please help, thank you.

Re: NiFi's GetHDFS processor with Cron schedule not reading all files in the directory

Schandhok — Mon, 26 Jun 2017 17:08:13 GMT

@Raj B This looks similar to NIFI-4069

As a workaround, please try and change the cron schedule to 0,30 30 0 * *. so that it runs twice in the same minute.

Let us know if that helps.

Re: NiFi's GetHDFS processor with Cron schedule not reading all files in the directory

Raj_B — Mon, 26 Jun 2017 20:41:23 GMT

@Shashank Chandhok the schedule change to "0,30 30 0 * * ?" helped to read few additional files, but many files still remain in the directory

Re: NiFi's GetHDFS processor with Cron schedule not reading all files in the directory

Schandhok — Mon, 26 Jun 2017 21:36:09 GMT

@Raj B

Please check the timestamps of the files remaining in the directory. If they are being added during the process run time. Or if the timestamp is older than the CRON runtime of the processor.

Re: NiFi's GetHDFS processor with Cron schedule not reading all files in the directory

Raj_B — Mon, 26 Jun 2017 22:06:26 GMT

@Shashank Chandhok actually, the files I'm trying to process are from the day before; in my directory path in GetHDFS processor, I'm using expression language to point to the directory that was created yesterday and the files in that directory are from yesterday. So when the CRON scheduler starts at 12:30 am, all files that would need to be processed should all be there already in that directory.

Re: NiFi's GetHDFS processor with Cron schedule not reading all files in the directory

Raj_B — Mon, 26 Jun 2017 22:09:27 GMT

Not sure why I need to schedule the GetHDFS processor to run continuously (I set to run every 15 seconds), but this schedule exhausts all files from the directory - 0/15 * * * * ?

In my case since I'm loading files the next day (GetHDFS directory path points to previous day's directory), this resolves the issue I was facing.

Re: NiFi's GetHDFS processor with Cron schedule not reading all files in the directory

Raj_B — Thu, 29 Jun 2017 00:20:05 GMT

Thanks to @Bryan Bende, I needed to change the batch size property in GetHDFS, to read all files in the directory.

https://community.hortonworks.com/questions/108547/need-clarification-on-how-nifi-processors-run-with.html#answer-109798