Support Questions

Find answers, ask questions, and share your expertise

How to maintain the moved file structure in NiFi FetchFile processor

avatar
Explorer

I have few files as follows.

/source/abc.txt

/source/xyz.txt

/source/subdir/abc-sub.txt

/source/subdir/xyz-sub.txt

 

Currently, all the files are copied to the output dir

/output/abc.txt

/output/xyz.txt

/output/abc-sub.txt

/output/xyz-sub.txt

 

Is it possible to re-create/maintain the original file hierarchy, as follows;

/output/abc.txt

/output/xyz.txt

/output/subdir/abc-sub.txt

/output/subdir/xyz-sub.txt

 

 

1 ACCEPTED SOLUTION

avatar
Super Guru

@JonnyL   The sample above provided exact output you requested.  Please inspect the template and inspect FlowFiles for root files and subdirectories files.   The path attribute, when the file comes from a subfolder is the subfolder ("/subdir") as needed.  For any files in root folder, then path is "./".    

 

Additionally the absolute path attribute is also there for all the files.  This attribute can be acted against too with deeper expression language, but I found path was easier.

View solution in original post

3 REPLIES 3

avatar
Super Guru

@JonnyL after FetchFile each flowfile contains an attribute called ${path}.  Use this path when writing the files like this:

 

PutFile SettingsPutFile Settings

 

Expected OutputExpected Output

 

 

You can find my template for this test here:

 

https://github.com/steven-dfheinz/NiFi-Templates/blob/master/ListFile_FetchFile_PutFile_Demo.xml

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

avatar
Explorer

I ran my flow and found the path attribute, but it is set to simply / so that is not a 100% solution.

 

My flow is based on the reply to this question https://community.cloudera.com/t5/Support-Questions/Hi-Everyone-Is-there-any-processor-available-in-...

 

However, subtly, I have a GetFile watching for a file dropped into a directory that kicks off the flow!. The name of the file is the same as that of the directory with .done, so I strip off the .done and then pass the directory name as the flowfile and continue as answer-186970...

 

avatar
Super Guru

@JonnyL   The sample above provided exact output you requested.  Please inspect the template and inspect FlowFiles for root files and subdirectories files.   The path attribute, when the file comes from a subfolder is the subfolder ("/subdir") as needed.  For any files in root folder, then path is "./".    

 

Additionally the absolute path attribute is also there for all the files.  This attribute can be acted against too with deeper expression language, but I found path was easier.