Using Hadoop Credential API to store AWS secrets
When running a distcp process from HDFS to AWS S3, credentials are required to authenticate to the S3 bucket. Passing these into the S3A URI would leak secret values into application logs. Storing these secrets in core-site.xml is also not ideal because this means any user with hdfs CLI access can access the S3 bucket to which these AWS credentials are tied.
The Hadoop Credential API can be used to manage access to S3 in a more fine-grained way.
The first step is to create a local JCEKS file in which to store the AWS Access Key and AWS Secret Key values:
hadoop credential create fs.s3a.access.key -provider localjceks://file/path/to/aws.jceks
<enter Access Key value at prompt>
hadoop credential create fs.s3a.secret.key -provider localjceks://file/path/to/aws.jceks
<enter Secret Key value at prompt>
We'll then copy this JCEKS file to HDFS with the appropriate permissions.
hdfs dfs -put /path/to/aws.jceks /user/admin/
hdfs dfs -chown admin:admin /user/admin/aws.jceks
hdfs dfs -chmod 400 /user/admin/aws.jceks
We can then use the credential provider when calling hadoop distcp, as follows:
hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/user/admin/aws.jceks /user/admin/file s3a://my-bucket/
Notice that only the admin user can read this credentials file. If other users attempt to run the command above they will receive a permissions error because they can't read aws.jceks.
This also works with hdfs commands, as in the below example.
hdfs dfs -Dhadoop.security.credential.provider.path=jceks://hdfs/user/admin/aws.jceks -ls s3a://my-bucket
