Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Unable to distcp from on premise hdfs to s3

avatar
Contributor

I am using this https://community.hortonworks.com/articles/59161/using-hadoop-credential-api-to-store-aws-secrets.ht... as a reference, but unable to successfully distcp or even -ls.

[root@test232 conf]# hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /nsswitch.conf s3a://kartik-test.s3-us-west-2.amazonaws.com

17/01/15 08:16:22 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:229) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:224) at org.apache.hadoop.tools.DistCp.run(DistCp.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:454) Invalid arguments: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist

I am now reading this: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html

but unable to understand what are the xml file names that need to be edited.

1 ACCEPTED SOLUTION

avatar
Contributor

This worked: [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /test s3a://kartik-test/

Thanks for all your help!!

View solution in original post

1 REPLY 1

avatar
Contributor

This worked: [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /test s3a://kartik-test/

Thanks for all your help!!