https://community.cloudera.com/t5/Archives-of-Support-Questions/Distcp-between-two-S3-clusters/m-p/224501#M69964 <P>Hi <A rel="user" href="https://community.cloudera.com/users/16846/brajeshreddy6.html" nodeid="16846">@Rajesh Reddy</A></P><P>I've tried it from S3 to a regular HDFS cluster, but I don't see why it wouldn't work. Instead of the hdfs:// prefix for the source/destinations, you can replace that with s3a:// If your S3a keys are defined elsewhere already, they don't necessarily need to be passed in-line like this example:</P><P>hadoop distcp -Dfs.s3a.access.key="&lt;key&gt;" -Dfs.s3a.secret.key="&lt;my Key&gt;" s3a://hostname/dir/file s3a://hostname/dir/file<BR /></P> Thu, 19 Oct 2017 22:59:01 GMT ssahi 2017-10-19T22:59:01Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Distcp-between-two-S3-clusters/m-p/224500#M69963 <P>Does distcp between two s3 clusters work? If yes, is it same as regular DistCp or how can it be achieved?</P> Thu, 19 Oct 2017 20:59:49 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Distcp-between-two-S3-clusters/m-p/224500#M69963 brajeshreddy 2017-10-19T20:59:49Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Distcp-between-two-S3-clusters/m-p/224501#M69964 <P>Hi <A rel="user" href="https://community.cloudera.com/users/16846/brajeshreddy6.html" nodeid="16846">@Rajesh Reddy</A></P><P>I've tried it from S3 to a regular HDFS cluster, but I don't see why it wouldn't work. Instead of the hdfs:// prefix for the source/destinations, you can replace that with s3a:// If your S3a keys are defined elsewhere already, they don't necessarily need to be passed in-line like this example:</P><P>hadoop distcp -Dfs.s3a.access.key="&lt;key&gt;" -Dfs.s3a.secret.key="&lt;my Key&gt;" s3a://hostname/dir/file s3a://hostname/dir/file<BR /></P> Thu, 19 Oct 2017 22:59:01 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Distcp-between-two-S3-clusters/m-p/224501#M69964 ssahi 2017-10-19T22:59:01Z