question How to maintain the moved file structure in NiFi FetchFile processor in Support Questions

How to maintain the moved file structure in NiFi FetchFile processor

JonnyL — Thu, 04 Jun 2020 10:35:55 GMT

I have few files as follows.

/source/abc.txt

/source/xyz.txt

/source/subdir/abc-sub.txt

/source/subdir/xyz-sub.txt

Currently, all the files are copied to the output dir

/output/abc.txt

/output/xyz.txt

/output/abc-sub.txt

/output/xyz-sub.txt

Is it possible to re-create/maintain the original file hierarchy, as follows;

/output/abc.txt

/output/xyz.txt

/output/subdir/abc-sub.txt

/output/subdir/xyz-sub.txt

Re: How to maintain the moved file structure in NiFi FetchFile processor

stevenmatison — Thu, 04 Jun 2020 12:10:11 GMT

@JonnyL after FetchFile each flowfile contains an attribute called ${path}. Use this path when writing the files like this:

PutFile Settings

Expected Output

You can find my template for this test here:

https://github.com/steven-dfheinz/NiFi-Templates/blob/master/ListFile_FetchFile_PutFile_Demo.xml

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

Thanks,

Steven @ DFHZ

Re: How to maintain the moved file structure in NiFi FetchFile processor

JonnyL — Thu, 04 Jun 2020 14:16:12 GMT

I ran my flow and found the path attribute, but it is set to simply / so that is not a 100% solution.

My flow is based on the reply to this question https://community.cloudera.com/t5/Support-Questions/Hi-Everyone-Is-there-any-processor-available-in-NiFi-to-read/m-p/219993?childToView=186970#answer-186970

However, subtly, I have a GetFile watching for a file dropped into a directory that kicks off the flow!. The name of the file is the same as that of the directory with .done, so I strip off the .done and then pass the directory name as the flowfile and continue as answer-186970...

Re: How to maintain the moved file structure in NiFi FetchFile processor

stevenmatison — Fri, 05 Jun 2020 12:20:33 GMT

@JonnyL The sample above provided exact output you requested. Please inspect the template and inspect FlowFiles for root files and subdirectories files. The path attribute, when the file comes from a subfolder is the subfolder ("/subdir") as needed. For any files in root folder, then path is "./".

Additionally the absolute path attribute is also there for all the files. This attribute can be acted against too with deeper expression language, but I found path was easier.