How to set HDFS replication with S3 connector

[#1]

Hello, I am not having any luck setting HDFS replication with the S3 connector in CM 5.16. I have tried using both IAM Role-based Authentication and Access Key Credentials but neither seem to work. There are no error messages but nothing from HDFS is being replicated. Has anyone else had success with HDFS replication to S3?

[#2]

Hi @JohnR ,

Thanks for reaching out to Cloudera community. 

Have you checked the below blog which contains some useful information on how to use Amazon S3 with BDR:

https://blog.cloudera.com/using-amazon-s3-with-cloudera-bdr/

Also, some public documentation is worthy to read too:

https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_auth_aws.html#concept_tmd_nsh_2y

and

https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_bdr_replication_intro.html

Thanks and hope this helps!

Li

[#3]

Thanks for the response. I have read over all of the provided links. 

Below are additional details:

• Configured external accounts using both Access Key Credentials and IAM role authentication.
• Neither works but I am not clear on what credentials need to be assigned to the IAM role or access key's account.

• Tried to setup a replication schedule for HDFS with each external account as the destination.
• Tried using "root" or root-like account on the underlying linux system(s) and "admin" from the cloudera gui.

• Tried both static and dynamic replication strategy

No errors appear on the dialog and no errors appear in cloudera events or logs.

Is there something else to check?