- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
nifi move json data between hdfs directories
- Labels:
-
Apache NiFi
Created on ‎10-21-2019 03:17 PM - edited ‎10-21-2019 03:28 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All,
I am moving data between hdfs directories to pick the latest updated flow file. The requirement is to move the merged json files. The code should verify on the source hdfs directories to pick the json files and process to the target along with the sub folders if the target directory doesn't have the directories available.
Source hdfs path:-
- /data/json/incoming/year=1970/month=01/day=18/$flow-file
Target hdfs path:-
- /data/json/final/$path/$flow-file
I am currently using listhdfs->updateattribute->movehdfs
In update attribute I have provided
property(filename): ${path}.substringAfterLast("/")
path: ${filename:substringBeforeLast('/')}
Movehdfs output directory has
Output Directory - /data/json/final/${path}
Could you please verify the flow and help to resolve the issue as I am unable to pull the files into target using movehdfs along with copying the sub-folders if it not already present.
Created ‎10-23-2019 07:51 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This has to be achieved only with puthdfs and not movehdfs.
The following nifi flow worked and issue resolved.
listhdfs->updateattribute->puthdfs->deletehdfs
Created ‎10-23-2019 07:51 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This has to be achieved only with puthdfs and not movehdfs.
The following nifi flow worked and issue resolved.
listhdfs->updateattribute->puthdfs->deletehdfs
