Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi Moving files on HDFS path for the previous day merged json file

Solved Go to solution
Highlighted

Nifi Moving files on HDFS path for the previous day merged json file

Explorer

I have a requirement to move the previous day processed and merged json files into new hdfs path. The requirement is to recursively search unprocessed files and move the pending unprocessed files.

Path 1 → /data/nifi/working/2019/10/source_2019_10_15.json — Daily processed files are merged under this path and gets added on daily basis.

Path 2 → /data/nifi/incoming/ — The code should search if folders doesn't exist then create and move the files are just move the files if the folders are already present.

 

Currently, I am using nifi flow -- ListHDFS→ MoveHDFS but unable to achieve it.

 

Need help how this can be achieved.

 

Thank you for the help.

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi Moving files on HDFS path for the previous day merged json file

Explorer

The current flow worked fine.

 listhdfs->fethchdfs->updateattribute->puthdfs->deletehdfs

 

In the list hdfs set the Minimum File Age wait time before consuming. This will allow the process to search files recursively.

 

I have completed all the activates except generating sequence number for each received flow file for the same date. Could you please check and help.

2 REPLIES 2

Re: Nifi Moving files on HDFS path for the previous day merged json file

Explorer

ListHDFS->RouteonAttribute->MoveHDFS

Can the above process flow be worked by utilizing RouteonAttribute to split and process previous day pending file to HDFS path?

Daily created Fileformat

stream_es_2019_10_14.json1
stream_es_2019_10_15.json1
stream_es_2019_10_16.json1

  1. If the filename is equal to today's date ignore to pick and process
  2. If the filename is lesser than today's date pick and move the file to defined folder path.

Please help if this can be achieved. 

 

 

Re: Nifi Moving files on HDFS path for the previous day merged json file

Explorer

The current flow worked fine.

 listhdfs->fethchdfs->updateattribute->puthdfs->deletehdfs

 

In the list hdfs set the Minimum File Age wait time before consuming. This will allow the process to search files recursively.

 

I have completed all the activates except generating sequence number for each received flow file for the same date. Could you please check and help.

Don't have an account?
Coming from Hortonworks? Activate your account here