Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

In NiFi how to write flowfiles into HDFS only if all mandatory files are received?

In NiFi how to write flowfiles into HDFS only if all mandatory files are received?

New Contributor

Hi,

I am expecting two files, file with abc.txt and file with pqr.txt my next processor should get triggered only if these two files are received.

Currently i am using listFile fetchFIle to capture the changes of source directory. Once i have received two mandatory files my next process is to process them. Now i am not sure how should i configure NiFi to hold the file in queue until i got all mandatory files.

Do we know how should we achieve this using NiFi?

2 REPLIES 2

Re: In NiFi how to write flowfiles into HDFS only if all mandatory files are received?

Master Guru

@RAUI

-

You may want to look in to using the "Wait" and "Notify" processors to accomplish this. The Wait processor would route a FlowFile to the wait relationship (default configuration) until a release signal is written to the cache service by the notify processor. So you route one file to wait and the other to Notify.

-

Do you know what order the files will always arrive in? If files arrive in no specific order, you may need two wait/notify sets of processors. One flow path has a wait feeding a notify and the other a notify feeding a wait. That will result in a wait occurring on either FlowFile no matter which file is received first.

-

Thanks,

Matt

-

When an "Answer" addresses/solves your question, please select "Accept" beneath that answer. This encourages user participation in this forum.

Re: In NiFi how to write flowfiles into HDFS only if all mandatory files are received?

New Contributor

@Matt Clarke, thanks for your answer, i am not aware about the file sequences.