Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

mapping a file to a path and routing it based on new filepath

mapping a file to a path and routing it based on new filepath

New Contributor

I am trying to transfer files from s3 bucket to another s3 bucket. I have to get the file path and map it to some other path and place the file to new path.

for ex.

s3://bucket-source/abc/xyz/20190225/fname.csv.gz has to put into s3://bucket-dest/xyz/ab/c/20190225/fname.csv.gz.

s3://bucket-source/ab1/xy1/2019/02/25/file2.csv.gz has to put into s3://bucket-dest/au/bc/2019/02/25/file2.csv.gz.

What is tried is :

ListS3 ---> FetchS3Object ---> ExecuteStreamCommand to call a python script with filename as argument, which is writing newfilepath in a text file. ---> FetchFile ---> ExtractText (new_filename=(.*)) ---> updateAttribute where I am replacing ${filename} = ${new_filename} ---> PutS3Object.

Here I am facing the problem, my file content is being replaced with the content of FetchFile file content.

Please let me know how can I make it work. Thanks in advance.

Don't have an account?
Coming from Hortonworks? Activate your account here