Support Questions

Find answers, ask questions, and share your expertise

Files in database are following Folder structure.But I am getting only Files. Is there any way to Fetch the Files with Folders, means it will store the data same as database(Folder inside folder inside file)

avatar
Explorer
 
1 ACCEPTED SOLUTION

avatar
Master Mentor

@ANKIT PATEL

Not sure what NiFi processors you may be using in your dataflow here, but take a look at the FlowFile attributes being created on the FlowFiles containing the files from your Database. You could likely use one of those attributes to re-create the directory structure you are looking for when writing the files back out of NiFi.

Thank you,

Matt

View solution in original post

5 REPLIES 5

avatar
Master Mentor

@ANKIT PATEL

Not sure what NiFi processors you may be using in your dataflow here, but take a look at the FlowFile attributes being created on the FlowFiles containing the files from your Database. You could likely use one of those attributes to re-create the directory structure you are looking for when writing the files back out of NiFi.

Thank you,

Matt

avatar
Explorer

I am not getting what you explained can you please elaborate?
for database connection, I have used GETFTP processor.

avatar
Master Mentor

@ANKIT PATEL

A NiFi FlowFile (this is what moves from processor to processor in NiFi) consists of two parts. FlowFile content (Actual data) and FlowFile Attributes (Key/value metadata about the FLowFile). Different processors that create FlowFile generate different attributes which are assigned to the FlowFile. Attributes will contain some things like filename, fileSize, UUID, path, etc...

-

According to the documentation for the GetFTP processor the following FlowFile Attributes will be written on each FlowFile produced:

64762-screen-shot-2018-03-22-at-104745-am.png

-

The NiFi Expression Language (EL) can then be used to do things with these key/value pairs. For example: setting a specific target directory to write a FlowFile's content.
-

Assuming you are writing FlowFiles out of NiFi using say putFile:

64763-screen-shot-2018-03-22-at-105117-am.png

The above EL "${path}" will return the value from the flowfile attribute named "path" and use it as the target directory path to write the content.

You can even add to that path if you like:

for example:

/mynewdir/nifi/${path}

The above appends the value of path to the end of /mynewdir/nifi/...

Thank you,

Matt

avatar

@ANKIT PATEL

Here's a sample flow. For simplicity, instead of reding from FTP location, I am reading from local path.

64774-screen-shot-2018-03-23-at-10756-am.png

I have 3 processors in my flow.

  1. ListFile - Will list the files in the directory I passed in the Configuration tab. Similar to ListFTP, just local.
  2. FetchFile - Will fetch the files that I will mention. Again, similar to FetchFTP, but just local.
  3. PutFile - Will write the data.

If you pay attention, ListFile processor is giving me the list of the files in folder, and since the downstream processor is stopped, the flow files are queued up. So I went ahead and did a "List Queue" to see the flow files which are queued up. I saw something like this.

64775-screen-shot-2018-03-23-at-11342-am.png

These are called as "flow files" in NiFi. If you click the "i" button to the leftmost side, you will see Attributes tab as shown below.

64776-screen-shot-2018-03-23-at-11526-am.png

64777-screen-shot-2018-03-23-at-125515-am.png

You can see many attributes, but the main attributes that we need in this example are

  • absolute.path - The location from where the file is
  • filename - The name of the file

These are the properties regarding the data we are about to fetch.

Now FetchFile can read these files from the given directory and I can tell my FetchFile processor to read these files by using these "attributes" as shown below.

64778-screen-shot-2018-03-23-at-12019-am.png

Since I have the directory name information available in the form an attribute, I can use it while storing the data as well and hence mimicking the exact directory structure from the source.

Hope that helps!

avatar
@ANKIT PATEL

Did the answer help in the resolution of your query? Please close the thread by marking the answer as Accepted!