Cloudera Community
AaronFabbri
user rank
Cloudera Employee

Re: Unable to copy data from HDFS to S3

by Guru Guru in Support Questions
‎02-05-2019 10:32 AM
‎02-05-2019 10:32 AM
Hi @VenkateshB,   You may want to take a look at the doc here where provides some distcp examples: https://www.cloudera.com/documentation/enterprise/6/latest/topics/cdh_admin_distcp_data_cluster_migrate.html#distcp_and_s3   Thanks and hope it helps, Li ... View more

Re: Unable to move data to a S3 bucket using last ...

by Explorer in Support Questions
‎03-05-2018 03:38 AM
‎03-05-2018 03:38 AM
Hi Aaron! Thanks for answering.   At the end it wasn't a problem with Hadoop or the configuration (credentials were correct and config files deploy in all nodes). It was just that IT was blocking all the traffic to the private bucket. Even after asking them to allow those IPs it didn't work so I install CNLM in all nodes and specified the proxy using:   -Dfs.s3a.proxy.host="localhost" -Dfs.s3a.proxy.port="3128"   After that I was able to move 3 TB in less than a day. ... View more

Re: Authentication for Amazon s3 using Temporary c...

by Cloudera Employee in Support Questions
‎02-01-2018 10:57 PM
‎02-01-2018 10:57 PM
STS should work. I would try (1) using s3a, not s3n, and (2) building your spark app with the same AWS SDK version as used in the cluster. ... View more
Contact Me
Online
Offline
Last Visited
‎05-15-2018 05:15 PM
Community Statistics
Member Since ‎09-09-2015 02:47 PM
Last Visited ‎05-15-2018 05:15 PM
Posts 5
Kudos received 2