Unable to distcp from on premise hdfs to s3

ddolecki — Sun, 15 Jan 2017 21:58:56 GMT

I am using this https://community.hortonworks.com/articles/59161/using-hadoop-credential-api-to-store-aws-secrets.html as a reference, but unable to successfully distcp or even -ls.

[root@test232 conf]# hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /nsswitch.conf s3a://kartik-test.s3-us-west-2.amazonaws.com

17/01/15 08:16:22 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:229) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:224) at org.apache.hadoop.tools.DistCp.run(DistCp.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:454) Invalid arguments: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist

I am now reading this: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html

but unable to understand what are the xml file names that need to be edited.

Re: Unable to distcp from on premise hdfs to s3

ddolecki — Sun, 15 Jan 2017 23:49:59 GMT

This worked: [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /test s3a://kartik-test/

Thanks for all your help!!

question Unable to distcp from on premise hdfs to s3 in Archives of Support Questions (Read Only)

Unable to distcp from on premise hdfs to s3

Re: Unable to distcp from on premise hdfs to s3