Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

Solved Go to solution

Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

You can simply run hdfs cli commands to delete objects on S3

hdfs dfs -rm -r -f -skipTrash s3a://S3_BUCKET_NAME/S3_PATH

This however will only works if you have specified S3 properties (fs.s3a.access.key, fs.s3a.secret.key) in core-site.xml

3 REPLIES 3

Re: Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

You can simply run hdfs cli commands to delete objects on S3

hdfs dfs -rm -r -f -skipTrash s3a://S3_BUCKET_NAME/S3_PATH

This however will only works if you have specified S3 properties (fs.s3a.access.key, fs.s3a.secret.key) in core-site.xml

Re: Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

so we're needing to perform the command inside an oozie workflow so am assuming the command above would work when the oozie workflow xml like what's shown below. Am I understanding that correctly?

<action name="remove-s3">

<fs>

<hdfs dfs -rm -r -f -skipTrash s3a://S3_BUCKET_NAME/S3_PATH />

</fs>

</action>

Re: Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

Yes, that is correct

Don't have an account?
Coming from Hortonworks? Activate your account here