How can we use a single set of Put/Get Processors to move multiple files to multiple directories?,Using Put/

[#1]

Here's my scenario: there are about 200+ files I need to move from Server A to Server B. Each file has a specific destination on Server B. I want to avoid creating 200+ sets of processors in order to move these files. I intend to create the exact same directory structure on Server B prior to the move.

So this becomes a 2-part question. 1: How do I use a single set of Get/Put processors to drop files onto the destination based on the file name. 2:(Optional) Is there a way to have Nifi create the directory structures automatically rather than doing so myself?

,

[#2 · Accepted solution]

@D'Andre McDonald

The Get based processors will create a "absolute.path" FlowFile attribute on all Files that are ingested in to NiFi.

So you would configure your Get processor to point at the base directory and consume files from all subdirectories.

The Put based processors support expression language in the "remote path" property. So you can use any attribute on the FlowFile to specify what path the file will be written to on the put. So here you could use ${absolute.path} as the value for this property.

The Put based processors also have a property for "create directory" which you can set to true.

Thank you,

Matt

[#3 · Accepted solution]

How are you planning to move the data between the servers? SFTP?

Lets say you have NiFi running on Server A, and you use GetFile to pick up the files from your directory, and PutSFTP to push the files to Server B. GetFile writes the "filename" and "absolute.path" on each flow file it picks up, and you can then reference those in the PutSFTP Directory property as "${absolute.path}/${filename}" and then set "Create Directory" to true. This assumes Server B is allowing SFPT.

[#4 · Accepted solution]

@D'Andre McDonald

The Get based processors will create a "absolute.path" FlowFile attribute on all Files that are ingested in to NiFi.

So you would configure your Get processor to point at the base directory and consume files from all subdirectories.

The Put based processors support expression language in the "remote path" property. So you can use any attribute on the FlowFile to specify what path the file will be written to on the put. So here you could use ${absolute.path} as the value for this property.

The Put based processors also have a property for "create directory" which you can set to true.

Thank you,

Matt

[#5 · Accepted solution]

How are you planning to move the data between the servers? SFTP?

Lets say you have NiFi running on Server A, and you use GetFile to pick up the files from your directory, and PutSFTP to push the files to Server B. GetFile writes the "filename" and "absolute.path" on each flow file it picks up, and you can then reference those in the PutSFTP Directory property as "${absolute.path}/${filename}" and then set "Create Directory" to true. This assumes Server B is allowing SFPT.

[#6 · Accepted solution]

@D'Andre McDonald

Here is what I did , which is basically what is described above..i have 80 files going in to 17 folders on a different server.(hoping Nifi user has access to the remote folder).

list and fetch will be your GetFile and then in my first UpdateAttribute I am extracting the filename (since my filename has other info which I am not interested in)..my destination folder on the remote system depends on my file name.

if you can route your files based on your incoming file name as it is you can ignore this step .

incoming file name to first UpdateAttribute is

export-20150218000000-20150219000000-FLGFUP04_LIQ_DA_ASP_VLV_DDR_F_CV

Out going file name is FLGFUP04_LIQ_DA_ASP_VLV_DDR_F_CV

my second UpdateAttribute process is where I set the Folder based on file name..here I have used Advanced tab but you can use properties also. basically I am wrote all my rules here..setting the destination Folder based on file names.

and finally in my in your put file you can use that property and create the folder dynamically..

Hope this helps...

Sai

[#7]

Very nice. Thanks

[#8]

@Matt: That makes perfect sense. Thanks!

@Bryan Bende: Thanks! Have the specifics on the properties that need to be set is the jumpstart I needed.

It sounds like I need to let the Put handle the "dynamic" part of this.

[#9]

@Saikrishna Tarapareddy : This actually answers what I had not yet asked concerning dynamic file names. Thanks much for the very detailed description! I began down this path when I tried to figure it out on my own but did not quite understand how to tie things together, until now. With your step-by-step I should be able to do this for the other scenario where I have various paths that are not nested under a single directory.