How to maintain the moved file structure in NiFi FetchFile processor

Question

I have few files as follows./source/abc.txt/source/xyz.txt/source/subdir/abc-sub.txt/source/subdir/xyz-sub.txt Currently, all the files are copied to the output dir/output/abc.txt/output/xyz.txt/output/abc-sub.txt/output/xyz-sub.txt Is it possible to re-create/maintain the original file hierarchy, as follows;/output/abc.txt/output/xyz.txt/output/subdir/abc-sub.txt/output/subdir/xyz-sub.txt

stevenmatison · Accepted Answer

@JonnyL The sample above provided exact output you requested. Please inspect the template and inspect FlowFiles for root files and subdirectories files. The path attribute, when the file comes from a subfolder is the subfolder ("/subdir") as needed. For any files in root folder, then path is "./".

Additionally the absolute path attribute is also there for all the files. This attribute can be acted against too with deeper expression language, but I found path was easier.

stevenmatison · Answer

@JonnyL after FetchFile each flowfile contains an attribute called ${path}. Use this path when writing the files like this:

PutFile Settings

Expected Output

You can find my template for this test here:

https://github.com/steven-dfheinz/NiFi-Templates/blob/master/ListFile_FetchFile_PutFile_Demo.xml

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

Thanks,

Steven @ DFHZ

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.