question Re: distcp from hdp 2.1 to s3a. in Archives of Support Questions (Read Only)

distcp from hdp 2.1 to s3a.

ashneesharma88 — Fri, 16 Sep 2022 10:45:53 GMT

Hi I am using HDP 2.1 and trying to do distcp from hdp 2.1 to s3a.

I configured hte access key and secret key. But getting error:-

[hive@test-reair ~]$ hadoop fs -ls s3a://test-poc/

ls: No FileSystem for scheme: s3a

Urgent help needed.

Thanks in advance.

myoung — Thu, 27 Oct 2016 20:44:15 GMT

ashneesharma88 — Fri, 28 Oct 2016 13:34:26 GMT

@ Michael Young

I have already gone through with this link, but not helped me.

myoung — Sat, 29 Oct 2016 02:30:56 GMT

Can you try using s3n instead of s3a? Support for s3a came into Hadoop with version 2.6.0 (https://issues.apache.org/jira/browse/HADOOP-10400). HDP 2.5 provides s3a as a Tech Preview. HDP 2.1 came with Hadoop 2.4.0 (http://hortonworks.com/blog/announcing-hdp-2-1-general-availability/)

ashneesharma88 — Wed, 02 Nov 2016 15:51:40 GMT

Hi @Michael Young

I have two cluster on one it is working but on second cluster after done the same it is not working with s3a.

ashneesharma88 — Tue, 08 Nov 2016 15:58:06 GMT

I have tried with s3n. It is working fine with s3n.

myoung — Tue, 08 Nov 2016 20:43:57 GMT

Good to know, thank you.