What is the best way to secure S3A objects on HDP 2.5?

[#1 · Accepted solution]

@eorgad

To protect the S3A access/secret keys, it is recommended that you use either:

1. IAM role-based authentication (such as EC2 instance profile), or
2. the Hadoop Credential Provider Framework - securely storing them and accessing them through configuration.

The Hadoop Credential Provider Framework allows secure "Credential Providers" to keep secrets outside Hadoop configuration files, storing them in encrypted files in local or Hadoop filesystems, and including them in requests. The Hadoop-AWS Module documentation describes how to configure this properly.