Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Move file to some other location after reading it using GetFile Processor

Hi,

I am reading a file using GetFile. If I set KeepSourceFile attribute to true then it keep on reading the file so I reset it to false. After resetting it to false, it removes the file from source after reading it. Is there any way to move the file to some other location once it has been read by GetFile?

15 REPLIES 15

@Muhammad Umar

The FetchFile processor will do what you want to do. The GetFile processor does not have the option of moving a file after reading it.

Just set the Completion Strategy to Move and then specify the directory in the Move Destination Directory

15422-screen-shot-2017-05-16-at-80716-am.png

@Wynner

I replaced the GetFile with FetchFile but its giving me a warning. Below is the warning:

Upstream connection is invalid because processor requires an upstream connection but currently it has no upstream connection.

Below is my data flow:

15420-fetchfile.png

@Muhammad Umar

All Fetch type processors require a List type processor preceding them to provide the files to fetch. So your FetchFile processor needs a list of files to fetch. So, put a ListFile processor configured to use the same directory as your FetchFile processor and send the output from the ListFile processor to the FetchFile processor.

Here is a link to an article that uses the ListSFTP and GetSFTP processors to explain the method: List to Fetch example in NiFi

@Wynner

I have added ListFile and FetchFile processors in my flow but they are not fetching the file.

Below are the configuration for both of the files:

ListFile:

15482-listfile.png

FetchFile:

15483-fetchfile.png

There is no errors or warnings in Nifi Log.

@Muhammad Umar

If you only want to pull that one file, then just configure ListFile this way

15501-screen-shot-2017-05-17-at-82824-am.png

then configure FetchFile this way

15502-screen-shot-2017-05-17-at-83000-am.png

It will get the one file and move it.

@Wynner,

I have successfully moved the file to HDFS using ListFile and Fetch as per your suggestions. But still there is one problem, although List file and Fetch file are moving my file from source to HDFS but List File throws an error and below is the error detail:

 WARN [Timer-Driven Process Thread-7] o.a.nifi.processors.standard.ListFile ListFile[id=1ae3923c-015c-1000-15e4-e29f5b75a0e0] Error collecting attributes for file /root/InputData/, message is Mount point not found

Other than this error, I need to ask one more thing; Is there any way to clear the listing of Listfile so that I can process the same file in future ?

@Muhammad Umar

What you want to do is to clear the state of the processor.

Right click on the ListFile processor, from the menu that pops up, select View state

15535-screen-shot-2017-05-18-at-81055-am.png

Then you will see a new window, in the middle right side of the window is a link Clear state, click that and the file will be picked up again the next time you want to pull it, and there you go.

15536-screen-shot-2017-05-18-at-81219-am.png

FYI, if you only want to pull the file once a day, you can configure the ListFile processor with a CRON driven Scheduling strategy and it will only run once per day.

@Wynner

I have already tried by clearing the state of processor and it did'nt work.

@Muhammad Umar

Was the processor stopped when you tried?

@Wynner

Yes..It was stopped.

@Muhammad Umar

Was there are error? What do you see after you click Clear state?

@Wynner

There is no error when I clear the state. When I click the "Clear State" link it removes the time-stamp etc entries below is the state after clearing:

15550-clear-state.png

Error only occurs when I run the process. And below is the error:

WARN [Timer-Driven Process Thread-7] o.a.nifi.processors.standard.ListFile ListFile[id=1ae3923c-015c-1000-15e4-e29f5b75a0e0] Error collecting attributes for file /root/InputData/, message is Mount point not found

clear-state.png

@Muhammad Umar

So, the Clear state is working.

The warning you are seeing has nothing to do with the state of the processor.

@Wynner

Yes.. But how can I remove this warning and why its occuring?

@Muhammad Umar

What is the OS of the system? How is the partition mounted, xfs, ext4?

Is this a VM or a bare metal server?

What is returned when you run the command "mount"?