Created 12-14-2017 08:59 AM
Hi Team, we noticed ListS3 processor is processing same file twice or more, while there was no change in the source file.
As we go though the "Component State" which clearly says, nifi manages the file states with latest time-stamp and list only when a new file has been added or existing file has been modified.
We are seeing ListS3 processor is reprocessing same file from Data Provenance which has the same s3.lastModifed property value, We believe file should not be re-processed unless its changed per "State Management" docs.
Kindly help us on this issue. Let us know for any information needed.
Note: ListS3 Processor has been setup to run only primary node.
Created 12-21-2017 01:17 AM
This is bug. There is an open defect also the workaround solutions.
Created 12-21-2017 06:08 AM
Thank You @Milan Das. So the workaround you suggested require changes to nifi code ?
For now I had added a DetectDuplicate processor.
Created 12-21-2017 11:59 AM
Yes a code fix in NIFI is needed. Work around is DetectDuplicate processor.