How to Know in which directory data stores after the process is completed in Nifi

[#1]

Hi All,

I created one process executed successfully GetFile operation , How to find in which directory the file is stored.

steps i followed

1.Opened web UI localhost:8080/nifi/

2.Getfile Processor

3.Linked Processor with log attribute with success relation

4.start process

5.File is moved from local file system to Nifi Directory

how to find that one in which folder it was stored.

[#2 · Accepted solution]

Typically the GetFile processor it pulls it into NiFi so you can do some type of processing or routing. It doesn't really put the file anywhere in particular.

You should use something like the PutFile processor to move the file to a location of your choosing. Just make sure to route the success relationship to the PutFile processor and configure the PutFile processing to your liking.

GetFile

PutFile

[#3]

Thank you Zblaco

[#4]

@AnjiReddy Anumolu

Just to add a little more detail to the above response from @zblanco.

When NiFi ingest data, that data is turned in to NiFi FlowFiles. A NiFi FlowFile consists of Attributes (Metadata) about the actual data and the physical data. The FlowFile metadata is stored in the FlowFile repository as well as JVM heap memory for faster performance. The FlowFile Attributes includes things like filename, ingest time, lineage age, filesize, what connection the FlowFile currently resides in dataflow, any user defined metadata, or processor added metadata, etc....). The physical bytes that make up the actual data content is written to claims within the NiFi content repository. A claim can contain the bytes for 1 to many ingest data files. For more info on the content repository and how claims work, see the following link:

https://community.hortonworks.com/articles/82308/understanding-how-nifis-content-repository-archivi....

Thanks,

Matt

[#5]

Normally The Content Repository holds the content for all the FlowFiles in the system. By default, it is installed in the same root installation directory as all the other repositories; as a admin you can configure it on a separate drive if available. e.g. check your {nifi_install_dir}/content_repository for contents.

[#6]

Thank you @milind pandit

[#7]

Hi @AnjiReddy Anumolu,

Easy way to get hold of your file is from provenance:

- on NiFi UI, click provenance button on top right corner

- find the event for your file, click on "view details" button

- you can view or download the file on the "contents" tab:

if you need to see the file contents on your server, search in the content_repository for file named as "identifier" from output claim [ie 1467063966583-11 as in screenshot above] @ "offset" [ie 463775 as in screenshot above] .

Hope this helps!

Thanks!

[#8]

Thank you @Jobin George

[#9]

Both of the above answers are correct. Just to provide a full picture of what is happening...

GetFile picks up the file from directory and brings it into NiFI's content_repository, which as milind pointed out is by default located under {nifi_install_dir}/content_repository. This directory is not meant to be used by the user, it is for NiFi's internal purposes.

The FlowFile is then transferred to LogAttribute which logs information, and I assume if that is the end of your flow then you must have marked the success relationship on LogAttribute as auto-terminated. At this point the flow file is removed from NiFi and the content in the content repository will eventually be removed.

NiFi is not meant to be a storage system where you bring data in and then leave it there, your flow would have to send the data somewhere after GetFile.

[#10]

Thank you @Bryan Bende