Created 03-22-2018 11:51 AM
Created 03-22-2018 12:27 PM
Not sure what NiFi processors you may be using in your dataflow here, but take a look at the FlowFile attributes being created on the FlowFiles containing the files from your Database. You could likely use one of those attributes to re-create the directory structure you are looking for when writing the files back out of NiFi.
Thank you,
Matt
Created 03-22-2018 12:27 PM
Not sure what NiFi processors you may be using in your dataflow here, but take a look at the FlowFile attributes being created on the FlowFiles containing the files from your Database. You could likely use one of those attributes to re-create the directory structure you are looking for when writing the files back out of NiFi.
Thank you,
Matt
Created 03-22-2018 02:14 PM
I am not getting what you explained can you please elaborate?
for database connection, I have used GETFTP processor.
Created on 03-22-2018 02:55 PM - edited 08-17-2019 11:27 PM
A NiFi FlowFile (this is what moves from processor to processor in NiFi) consists of two parts. FlowFile content (Actual data) and FlowFile Attributes (Key/value metadata about the FLowFile). Different processors that create FlowFile generate different attributes which are assigned to the FlowFile. Attributes will contain some things like filename, fileSize, UUID, path, etc...
-
According to the documentation for the GetFTP processor the following FlowFile Attributes will be written on each FlowFile produced:
-
The NiFi Expression Language (EL) can then be used to do things with these key/value pairs. For example: setting a specific target directory to write a FlowFile's content.
-
Assuming you are writing FlowFiles out of NiFi using say putFile:
The above EL "${path}" will return the value from the flowfile attribute named "path" and use it as the target directory path to write the content.
You can even add to that path if you like:
for example:
/mynewdir/nifi/${path}
The above appends the value of path to the end of /mynewdir/nifi/...
Thank you,
Matt
Created on 03-23-2018 05:24 AM - edited 08-17-2019 11:27 PM
Here's a sample flow. For simplicity, instead of reding from FTP location, I am reading from local path.
I have 3 processors in my flow.
If you pay attention, ListFile processor is giving me the list of the files in folder, and since the downstream processor is stopped, the flow files are queued up. So I went ahead and did a "List Queue" to see the flow files which are queued up. I saw something like this.
These are called as "flow files" in NiFi. If you click the "i" button to the leftmost side, you will see Attributes tab as shown below.
You can see many attributes, but the main attributes that we need in this example are
These are the properties regarding the data we are about to fetch.
Now FetchFile can read these files from the given directory and I can tell my FetchFile processor to read these files by using these "attributes" as shown below.
Since I have the directory name information available in the form an attribute, I can use it while storing the data as well and hence mimicking the exact directory structure from the source.
Hope that helps!
Created 04-01-2018 04:23 PM
Did the answer help in the resolution of your query? Please close the thread by marking the answer as Accepted!