Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

avatar
 
1 ACCEPTED SOLUTION

avatar

You can simply run hdfs cli commands to delete objects on S3

hdfs dfs -rm -r -f -skipTrash s3a://S3_BUCKET_NAME/S3_PATH

This however will only works if you have specified S3 properties (fs.s3a.access.key, fs.s3a.secret.key) in core-site.xml

View solution in original post

3 REPLIES 3

avatar

You can simply run hdfs cli commands to delete objects on S3

hdfs dfs -rm -r -f -skipTrash s3a://S3_BUCKET_NAME/S3_PATH

This however will only works if you have specified S3 properties (fs.s3a.access.key, fs.s3a.secret.key) in core-site.xml

avatar

so we're needing to perform the command inside an oozie workflow so am assuming the command above would work when the oozie workflow xml like what's shown below. Am I understanding that correctly?

<action name="remove-s3">

<fs>

<hdfs dfs -rm -r -f -skipTrash s3a://S3_BUCKET_NAME/S3_PATH />

</fs>

</action>

avatar

Yes, that is correct