Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can we use a single set of Put/Get Processors to move multiple files to multiple directories?,Using Put/

avatar
New Contributor

Here's my scenario: there are about 200+ files I need to move from Server A to Server B. Each file has a specific destination on Server B. I want to avoid creating 200+ sets of processors in order to move these files. I intend to create the exact same directory structure on Server B prior to the move.

So this becomes a 2-part question. 1: How do I use a single set of Get/Put processors to drop files onto the destination based on the file name. 2:(Optional) Is there a way to have Nifi create the directory structures automatically rather than doing so myself?

,
3 ACCEPTED SOLUTIONS

avatar
Master Mentor

@D'Andre McDonald

The Get based processors will create a "absolute.path" FlowFile attribute on all Files that are ingested in to NiFi.

So you would configure your Get processor to point at the base directory and consume files from all subdirectories.

The Put based processors support expression language in the "remote path" property. So you can use any attribute on the FlowFile to specify what path the file will be written to on the put. So here you could use ${absolute.path} as the value for this property.

The Put based processors also have a property for "create directory" which you can set to true.

Thank you,

Matt

View solution in original post

avatar
Master Guru

How are you planning to move the data between the servers? SFTP?

Lets say you have NiFi running on Server A, and you use GetFile to pick up the files from your directory, and PutSFTP to push the files to Server B. GetFile writes the "filename" and "absolute.path" on each flow file it picks up, and you can then reference those in the PutSFTP Directory property as "${absolute.path}/${filename}" and then set "Create Directory" to true. This assumes Server B is allowing SFPT.

View solution in original post

avatar
Super Collaborator

@D'Andre McDonald

Here is what I did , which is basically what is described above..i have 80 files going in to 17 folders on a different server.(hoping Nifi user has access to the remote folder).

10561-folder.png

list and fetch will be your GetFile and then in my first UpdateAttribute I am extracting the filename (since my filename has other info which I am not interested in)..my destination folder on the remote system depends on my file name.

if you can route your files based on your incoming file name as it is you can ignore this step .

incoming file name to first UpdateAttribute is

export-20150218000000-20150219000000-FLGFUP04_LIQ_DA_ASP_VLV_DDR_F_CV

Out going file name is FLGFUP04_LIQ_DA_ASP_VLV_DDR_F_CV

10562-tbjh3.png

my second UpdateAttribute process is where I set the Folder based on file name..here I have used Advanced tab but you can use properties also. basically I am wrote all my rules here..setting the destination Folder based on file names.

10563-jqmf7.png

and finally in my in your put file you can use that property and create the folder dynamically..

10564-lkr8h.png

Hope this helps...

Sai

View solution in original post

6 REPLIES 6

avatar
Master Mentor

@D'Andre McDonald

The Get based processors will create a "absolute.path" FlowFile attribute on all Files that are ingested in to NiFi.

So you would configure your Get processor to point at the base directory and consume files from all subdirectories.

The Put based processors support expression language in the "remote path" property. So you can use any attribute on the FlowFile to specify what path the file will be written to on the put. So here you could use ${absolute.path} as the value for this property.

The Put based processors also have a property for "create directory" which you can set to true.

Thank you,

Matt

avatar
Master Guru

How are you planning to move the data between the servers? SFTP?

Lets say you have NiFi running on Server A, and you use GetFile to pick up the files from your directory, and PutSFTP to push the files to Server B. GetFile writes the "filename" and "absolute.path" on each flow file it picks up, and you can then reference those in the PutSFTP Directory property as "${absolute.path}/${filename}" and then set "Create Directory" to true. This assumes Server B is allowing SFPT.

avatar
Super Collaborator

@D'Andre McDonald

Here is what I did , which is basically what is described above..i have 80 files going in to 17 folders on a different server.(hoping Nifi user has access to the remote folder).

10561-folder.png

list and fetch will be your GetFile and then in my first UpdateAttribute I am extracting the filename (since my filename has other info which I am not interested in)..my destination folder on the remote system depends on my file name.

if you can route your files based on your incoming file name as it is you can ignore this step .

incoming file name to first UpdateAttribute is

export-20150218000000-20150219000000-FLGFUP04_LIQ_DA_ASP_VLV_DDR_F_CV

Out going file name is FLGFUP04_LIQ_DA_ASP_VLV_DDR_F_CV

10562-tbjh3.png

my second UpdateAttribute process is where I set the Folder based on file name..here I have used Advanced tab but you can use properties also. basically I am wrote all my rules here..setting the destination Folder based on file names.

10563-jqmf7.png

and finally in my in your put file you can use that property and create the folder dynamically..

10564-lkr8h.png

Hope this helps...

Sai

avatar
New Contributor

Very nice. Thanks

avatar
New Contributor

@Matt: That makes perfect sense. Thanks!

@Bryan Bende: Thanks! Have the specifics on the properties that need to be set is the jumpstart I needed.

It sounds like I need to let the Put handle the "dynamic" part of this.

avatar
New Contributor

@Saikrishna Tarapareddy : This actually answers what I had not yet asked concerning dynamic file names. Thanks much for the very detailed description! I began down this path when I tried to figure it out on my own but did not quite understand how to tie things together, until now. With your step-by-step I should be able to do this for the other scenario where I have various paths that are not nested under a single directory.