Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

How to configure HDP2.6 to use S3?

Expert Contributor


I am looking to use S3 instead of HDFS to store data and run computation on my Cluster (build on EC2 instances) .


1. How can I configure HDP to use AWS IAM role to interact with S3? I dont want to use AWS keys to interact with S3

2. Is S3 Guard available in HDP2.6?

3. Any best practices to follow when using S3 instead of HDFS?




Expert Contributor
@Pradeep Bhadani

If you would like to use Cloudbreak to provision your HDP stack then here are the answers:

1. -> Instance Profile option

2. this is in TP in HDP 2.6

3. a documentation regarding S3 connector



Expert Contributor

@rdoktorics Thanks for the link.

But I am looking to install HDP manually (Not via Cloudbreak or HDC).

Expert Contributor

@Pradeep Bhadani

You have to attach an AWS instance profile to every machine which you would like to use with s3. After that you are able to reach s3 with s3a://



Expert Contributor

@rdoktorics I have attached AWS Instance Profile to all EC2 machines and i can successfully use aws cli to list/get S3 bucket but i cannot run command hdfs dfs -ls <S3_BUCKET_PATH>

Question here is how to make hadoop use the AWS instance profile. What all properties are required to add/set in core-site.xml or hdfs-site.xml?

Expert Contributor

I see way to do in Hadoop 2.8 but HDP ships Hadoop 2.7.x

Expert Contributor

I added following property and able to list S3 bucket from hdfs comamnd

But not sure if this property is sufficient when interacting with S3.

Expert Contributor

In this documentation:

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.