Files in database are following Folder structure.But I am getting only Files. Is there any way to Fetch the Files with Folders, means it will store the data same as database(Folder inside folder inside file)

[#1 · Accepted solution]

@ANKIT PATEL

Not sure what NiFi processors you may be using in your dataflow here, but take a look at the FlowFile attributes being created on the FlowFiles containing the files from your Database. You could likely use one of those attributes to re-create the directory structure you are looking for when writing the files back out of NiFi.

Thank you,

Matt

[#2]

I am not getting what you explained can you please elaborate?
for database connection, I have used GETFTP processor.

[#3]

@ANKIT PATEL

A NiFi FlowFile (this is what moves from processor to processor in NiFi) consists of two parts. FlowFile content (Actual data) and FlowFile Attributes (Key/value metadata about the FLowFile). Different processors that create FlowFile generate different attributes which are assigned to the FlowFile. Attributes will contain some things like filename, fileSize, UUID, path, etc...

-

According to the documentation for the GetFTP processor the following FlowFile Attributes will be written on each FlowFile produced:

-

The NiFi Expression Language (EL) can then be used to do things with these key/value pairs. For example: setting a specific target directory to write a FlowFile's content.
-

Assuming you are writing FlowFiles out of NiFi using say putFile:

The above EL "${path}" will return the value from the flowfile attribute named "path" and use it as the target directory path to write the content.

You can even add to that path if you like:

for example:

/mynewdir/nifi/${path}

The above appends the value of path to the end of /mynewdir/nifi/...

Thank you,

Matt

[#4]

@ANKIT PATEL

Here's a sample flow. For simplicity, instead of reding from FTP location, I am reading from local path.

I have 3 processors in my flow.

1. ListFile - Will list the files in the directory I passed in the Configuration tab. Similar to ListFTP, just local.
2. FetchFile - Will fetch the files that I will mention. Again, similar to FetchFTP, but just local.
3. PutFile - Will write the data.

If you pay attention, ListFile processor is giving me the list of the files in folder, and since the downstream processor is stopped, the flow files are queued up. So I went ahead and did a "List Queue" to see the flow files which are queued up. I saw something like this.

These are called as "flow files" in NiFi. If you click the "i" button to the leftmost side, you will see Attributes tab as shown below.

You can see many attributes, but the main attributes that we need in this example are

• absolute.path - The location from where the file is
• filename - The name of the file

These are the properties regarding the data we are about to fetch.

Now FetchFile can read these files from the given directory and I can tell my FetchFile processor to read these files by using these "attributes" as shown below.

Since I have the directory name information available in the form an attribute, I can use it while storing the data as well and hence mimicking the exact directory structure from the source.

Hope that helps!

[#5]

@ANKIT PATEL

Did the answer help in the resolution of your query? Please close the thread by marking the answer as Accepted!