Support Questions
Find answers, ask questions, and share your expertise

How to set HDFS replication with S3 connector

Explorer

Hello, I am not having any luck setting HDFS replication with the S3 connector in CM 5.16. I have tried using both IAM Role-based Authentication and Access Key Credentials but neither seem to work. There are no error messages but nothing from HDFS is being replicated. Has anyone else had success with HDFS replication to S3?

2 REPLIES 2

Super Collaborator

Hi @JohnR ,

 

Thanks for reaching out to Cloudera community. 

 

Have you checked the below blog which contains some useful information on how to use Amazon S3 with BDR:

https://blog.cloudera.com/using-amazon-s3-with-cloudera-bdr/

 

Also, some public documentation is worthy to read too:

https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_auth_aws.html#concept_tmd_nsh_2y

and

https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_bdr_replication_intro.html

 

Thanks and hope this helps!

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Explorer

Thanks for the response. I have read over all of the provided links. 

Below are additional details:

  • Configured external accounts using both Access Key Credentials and IAM role authentication.
  • Neither works but I am not clear on what credentials need to be assigned to the IAM role or access key's account.

JohnR_3-1587756009479.png

 

  • Tried to setup a replication schedule for HDFS with each external account as the destination.
  • Tried using "root" or root-like account on the underlying linux system(s) and "admin" from the cloudera gui.

JohnR_4-1587756035432.png

  • Tried both static and dynamic replication strategy

JohnR_5-1587756388483.png

No errors appear on the dialog and no errors appear in cloudera events or logs.

 

Is there something else to check?