Support Questions
Find answers, ask questions, and share your expertise

Nifi-Dependency on two flowfiles to kick off a spark-submit

Highlighted

Nifi-Dependency on two flowfiles to kick off a spark-submit

New Contributor

Hi,

I am pretty new to Nifi and working out to create process flow for my application.

I have a process which waits for control files(.ctl) once they arrive the corresponding avro files are moved in to HDFS and there two kinds of files. lets say <file-type-a-timestamp> and <file-type-b-timestamp>.

Once i receive the corresponding <file-type-a-timestamp> and <file-type-b-timestamp> based on the timestamp, i have to kick of a spark application by passing both files as arguments with the hdfs location.

At a given point in time, i might have a more than one <file-type-a-timestamp> and/or <file-type-b-timestamp>.

Which processor can i use for this method ? As of now, i have created a group for waiting for control files and moving to HDFS and used a output port to send the information to other group, where i am planning for kicking of the spark application.

Thanks and appreciate your feedback and help.