NiFi: Cron Schedule not working as expected

@Jagapriyan 
Since you are using the Listing Strategy "Tracking Timestamps", the configuration property "Entity Tracking Time Window" is not used.  The "Tracking Timestamps" strategy is very dependent on timestamps of the target files. Typically when files are not being picked up it is because the timestamps on those files are equal to or less than the last recorded timestamp in the ListSFTP processors state.  This can happen when files in the SFTP server target folders do not have their last modified timestamp updated (for example moving a file from another directory into a SFTP server directory. A copy would update the timestamp since the file is being written again).

- Does your target SFTP path have multiple sub-directories which are being searched?  Is Search Recursively set to "true"? 
- Are there symlink directories in use?
- Have you looked the the state recorded timestamp for your SFTP server directories?  Are your missed files having older timestamps?
- How many files average are being written to the target SFTP between 12am and 1am each day?

I also see you have min file age of 5 minutes. This means the last Modified timestamp must be 5 minutes older than the execution time of your processor for the file to be eligible for consumption.   I see you stated your files are placed in the SFTP server between 12am - 1am each day and you scheduled your ListSFTP processor using a cron schedule at 10 minutes and 1 second every hour between 2am and 2pm.   Why not just have your listSFTP processor run all the time?  Is this because timestamps are not being updated consistently?

If you switch to using the listing strategy "Tracking Entities" instead, do you still see the issue? Tracking entities works when there is issues with timestamps and was developed for that reason.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Hi @MattWho 

Please find my responses

- Does your target SFTP path have multiple sub-directories which are being searched?  Is Search Recursively set to "true"?   --> Search recursively is set to true
- Are there symlink directories in use? - No
- Have you looked the state recorded timestamp for your SFTP server directories?  Are your missed files having older timestamps? -- Missing files have older time stamps. 
- How many files average are being written to the target SFTP between 12am and 1am each day? file count ranges from 10 - 100 and all these files are not being picked. 

Why not just have your listSFTP processor run all the time?  Is this because timestamps are not being updated consistently? Even running the processor all the time with cron schedule is not picking the files. 

Thanks @MattWho 

This has solved the problem.