Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

nifi move json data between hdfs directories

Solved Go to solution

nifi move json data between hdfs directories

Explorer

Hi All,

 

I am moving data between hdfs directories to pick the latest updated flow file. The requirement is to move the merged json files. The code should verify on the source hdfs directories to pick the json files and process to the target along with the sub folders if the target directory doesn't have the directories available.

Source hdfs path:-

  • /data/json/incoming/year=1970/month=01/day=18/$flow-file

Target hdfs path:-

  • /data/json/final/$path/$flow-file

I am currently using listhdfs->updateattribute->movehdfs

In update attribute I have provided 

property(filename): ${path}.substringAfterLast("/")

path: ${filename:substringBeforeLast('/')}

 

Movehdfs output directory has 

Output Directory - /data/json/final/${path}

 

Could you please verify the flow and help to resolve the issue as I am unable to pull the files into target using movehdfs along with copying the sub-folders if it not already present.

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: nifi move json data between hdfs directories

Explorer

This has to be achieved only with puthdfs and not movehdfs.

The following nifi flow worked and issue resolved.

 listhdfs->updateattribute->puthdfs->deletehdfs

1 REPLY 1

Re: nifi move json data between hdfs directories

Explorer

This has to be achieved only with puthdfs and not movehdfs.

The following nifi flow worked and issue resolved.

 listhdfs->updateattribute->puthdfs->deletehdfs

Don't have an account?
Coming from Hortonworks? Activate your account here