Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Compare filename attribute of flowfiles from fetchs3 and querydatabasetable processors

Highlighted

Compare filename attribute of flowfiles from fetchs3 and querydatabasetable processors

Contributor

Hi All,

Thanks a lot for the help.

I am stuck in a situation, I have lists3 and fetchs3 processors. fetchs3 brings the files from s3, there is a querydatabasetable processor which queries the table and brings in the list of filename already written in DB. Now we need to compare the filenames from fetchs3 with the filenames from querydatabasetable and only write those files which have not been written to table before.

My approach was using using detectduplicate processor, writing the result of querydatabasetable to detectduplicate and then compare it to fetchs3.

But I am not able to achieve it in NiFi, any suggestions ideas will help

Thanks

Dhieru