question Re: Nifi SFTP Mutiple Files into Mutiple HDFS directories (Based on file name) in Archives of Support Questions (Read Only)

Nifi SFTP Mutiple Files into Mutiple HDFS directories (Based on file name)

jpetro416 — Thu, 23 Feb 2017 05:55:54 GMT

Trying to move files via Nifi using GetSFTP. Let's say the filename is "DATA1". I want to move DATA1 into HDFS directory DATA1. I also want to do this with DATA2, DATA3, etc... How would this be possible via Nifi without creating a process for each file?

Re: Nifi SFTP Mutiple Files into Mutiple HDFS directories (Based on file name)

bbende — Thu, 23 Feb 2017 06:03:33 GMT

Use expression language in PutHDFS for the value of the Directory property, so setting it to ${filename} would put each file into a directory named the same as the filename attribute of each flow file in NiFi.

Re: Nifi SFTP Mutiple Files into Mutiple HDFS directories (Based on file name)

MattWho — Mon, 19 Aug 2019 10:34:13 GMT

@Joe Petro

Yes this is very doable...

NiFi automatically creates a FlowFile Attribute called "filename" on every FlowFile that is created. You can use this existing attribute to specify teh target HDFS directory:

Of course you will want to modify the above for the complete target path.

Thanks,

Matt

Re: Nifi SFTP Mutiple Files into Mutiple HDFS directories (Based on file name)

jpetro416 — Thu, 23 Feb 2017 06:16:29 GMT

Thanks, this is exactly what I'm looking for!