question Re: Files in database are following Folder structure.But I am getting only Files. Is there any way to Fetch the Files with Folders, means it will store the data same as database(Folder inside folder inside file) in Archives of Support Questions (Read Only)

Files in database are following Folder structure.But I am getting only Files. Is there any way to Fetch the Files with Folders, means it will store the data same as database(Folder inside folder inside file)

apatel750 — Thu, 22 Mar 2018 18:51:41 GMT

Re: Files in database are following Folder structure.But I am getting only Files. Is there any way to Fetch the Files with Folders, means it will store the data same as database(Folder inside folder inside file)

MattWho — Thu, 22 Mar 2018 19:27:55 GMT

Not sure what NiFi processors you may be using in your dataflow here, but take a look at the FlowFile attributes being created on the FlowFiles containing the files from your Database. You could likely use one of those attributes to re-create the directory structure you are looking for when writing the files back out of NiFi.

Thank you,

Matt

Re: Files in database are following Folder structure.But I am getting only Files. Is there any way to Fetch the Files with Folders, means it will store the data same as database(Folder inside folder inside file)

apatel750 — Thu, 22 Mar 2018 21:14:27 GMT

I am not getting what you explained can you please elaborate?
for database connection, I have used GETFTP processor.

Re: Files in database are following Folder structure.But I am getting only Files. Is there any way to Fetch the Files with Folders, means it will store the data same as database(Folder inside folder inside file)

MattWho — Sun, 18 Aug 2019 06:27:21 GMT

@ANKIT PATEL

A NiFi FlowFile (this is what moves from processor to processor in NiFi) consists of two parts. FlowFile content (Actual data) and FlowFile Attributes (Key/value metadata about the FLowFile). Different processors that create FlowFile generate different attributes which are assigned to the FlowFile. Attributes will contain some things like filename, fileSize, UUID, path, etc...

According to the documentation for the GetFTP processor the following FlowFile Attributes will be written on each FlowFile produced:

The NiFi Expression Language (EL) can then be used to do things with these key/value pairs. For example: setting a specific target directory to write a FlowFile's content.
-

Assuming you are writing FlowFiles out of NiFi using say putFile:

The above EL "${path}" will return the value from the flowfile attribute named "path" and use it as the target directory path to write the content.

You can even add to that path if you like:

for example:

/mynewdir/nifi/${path}

The above appends the value of path to the end of /mynewdir/nifi/...

Thank you,

Matt

Re: Files in database are following Folder structure.But I am getting only Files. Is there any way to Fetch the Files with Folders, means it will store the data same as database(Folder inside folder inside file)

RahulSoni — Sun, 18 Aug 2019 06:27:07 GMT

@ANKIT PATEL

Here's a sample flow. For simplicity, instead of reding from FTP location, I am reading from local path.

I have 3 processors in my flow.

ListFile - Will list the files in the directory I passed in the Configuration tab. Similar to ListFTP, just local.
FetchFile - Will fetch the files that I will mention. Again, similar to FetchFTP, but just local.
PutFile - Will write the data.

If you pay attention, ListFile processor is giving me the list of the files in folder, and since the downstream processor is stopped, the flow files are queued up. So I went ahead and did a "List Queue" to see the flow files which are queued up. I saw something like this.

These are called as "flow files" in NiFi. If you click the "i" button to the leftmost side, you will see Attributes tab as shown below.

You can see many attributes, but the main attributes that we need in this example are

absolute.path - The location from where the file is
filename - The name of the file

These are the properties regarding the data we are about to fetch.

Now FetchFile can read these files from the given directory and I can tell my FetchFile processor to read these files by using these "attributes" as shown below.

Since I have the directory name information available in the form an attribute, I can use it while storing the data as well and hence mimicking the exact directory structure from the source.

Hope that helps!

Re: Files in database are following Folder structure.But I am getting only Files. Is there any way to Fetch the Files with Folders, means it will store the data same as database(Folder inside folder inside file)

RahulSoni — Sun, 01 Apr 2018 23:23:04 GMT

@ANKIT PATEL

Did the answer help in the resolution of your query? Please close the thread by marking the answer as Accepted!