question Re: Nifi Moving files on HDFS path for the previous day merged json file in Support Questions

Nifi Moving files on HDFS path for the previous day merged json file

Deenag — Thu, 17 Oct 2019 01:39:12 GMT

I have a requirement to move the previous day processed and merged json files into new hdfs path. The requirement is to recursively search unprocessed files and move the pending unprocessed files.

Path 1 → /data/nifi/working/2019/10/source_2019_10_15.json — Daily processed files are merged under this path and gets added on daily basis.

Path 2 → /data/nifi/incoming/ — The code should search if folders doesn't exist then create and move the files are just move the files if the folders are already present.

Currently, I am using nifi flow -- ListHDFS→ MoveHDFS but unable to achieve it.

Need help how this can be achieved.

Thank you for the help.

Re: Nifi Moving files on HDFS path for the previous day merged json file

Deenag — Wed, 16 Oct 2019 20:58:33 GMT

ListHDFS->RouteonAttribute->MoveHDFS

Can the above process flow be worked by utilizing RouteonAttribute to split and process previous day pending file to HDFS path?

Daily created Fileformat

stream_es_2019_10_14.json1
stream_es_2019_10_15.json1
stream_es_2019_10_16.json1

If the filename is equal to today's date ignore to pick and process
If the filename is lesser than today's date pick and move the file to defined folder path.

Please help if this can be achieved.

Re: Nifi Moving files on HDFS path for the previous day merged json file

Deenag — Thu, 24 Oct 2019 18:17:20 GMT

The current flow worked fine.

listhdfs->fethchdfs->updateattribute->puthdfs->deletehdfs

In the list hdfs set the Minimum File Age wait time before consuming. This will allow the process to search files recursively.

I have completed all the activates except generating sequence number for each received flow file for the same date. Could you please check and help.