Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache Nifi to move files to new hdfs folder for the files lesser than current date

Highlighted

Apache Nifi to move files to new hdfs folder for the files lesser than current date

Explorer

am creating end to end flow to consume data into HDFS by using Consume Kafka for the Json files received through tealium event stream.

Currently, I have used

Consume Kafka -> Evaluate Json Path -> Jolttransform Json -> Merge Content -> Evaluate Json Path -> Update attribute -> PutHDFS ->MoveHDFS

 

The requirement is to read JSON Data for entire day spool into a single file referring to attribute postdate(covert epoch to YYYYMMDDSS timestamp before) and read data daily to merge into a single output file and finally rename the file as per the Timestamp related to POST_DATE field to differentiate daily files.

The current date output folder should have only current date processing files and all the completed output file for earlier dates should move to a different folder.

Could you please help me how to work on MoveHDFS to recursively search the hdfs folder and move completed output files not equal to current date to move into a different folder.

Don't have an account?
Coming from Hortonworks? Activate your account here