question Re: Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive") in Archives of Support Questions (Read Only)

Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

terri_ulrich — Wed, 21 Jun 2017 22:27:49 GMT

Re: Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

namaheshwari — Thu, 22 Jun 2017 01:29:44 GMT

You can simply run hdfs cli commands to delete objects on S3

hdfs dfs -rm -r -f -skipTrash s3a://S3_BUCKET_NAME/S3_PATH

This however will only works if you have specified S3 properties (fs.s3a.access.key, fs.s3a.secret.key) in core-site.xml

Re: Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

terri_ulrich — Thu, 22 Jun 2017 02:39:20 GMT

so we're needing to perform the command inside an oozie workflow so am assuming the command above would work when the oozie workflow xml like what's shown below. Am I understanding that correctly?

<fs>

</fs>

</action>

Re: Does anyone know how to execute aws cli fommand to remove s3 objects from within oozie workflow (would need to do something like this command within oozie workflow "aws s3 rm s3://PathToObjects --recursive")

namaheshwari — Thu, 22 Jun 2017 03:32:55 GMT

Yes, that is correct