Cloudera Community
ddolecki
user rank
Contributor
My Accepted Solutions
Show All

Re: Unable to distcp from on premise hdfs to s3

by Contributor in Support Questions
‎01-15-2017 03:49 PM
‎01-15-2017 03:49 PM
This worked: [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /test s3a://kartik-test/ Thanks for all your help!! ... View more

Re: How to install the hadoop-aws module to copy f...

by Contributor in Support Questions
‎01-15-2017 03:00 PM
‎01-15-2017 03:00 PM
This worked: [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /test s3a://kartik-test/ Thanks for all your help!! ... View more

Unable to distcp from on premise hdfs to s3

by Contributor in Support Questions
‎01-15-2017 01:58 PM
‎01-15-2017 01:58 PM
I am using this https://community.hortonworks.com/articles/59161/using-hadoop-credential-api-to-store-aws-secrets.html as a reference, but unable to successfully distcp or even -ls. [root@test232 conf]# hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /nsswitch.conf s3a://kartik-test.s3-us-west-2.amazonaws.com 17/01/15 08:16:22 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:229) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:224) at org.apache.hadoop.tools.DistCp.run(DistCp.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:454) Invalid arguments: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist I am now reading this: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html but unable to understand what are the xml file names that need to be edited. ... View more
Labels:

Re: How to install the hadoop-aws module to copy f...

by Contributor in Support Questions
‎01-15-2017 01:16 AM
‎01-15-2017 01:16 AM
Regenerated keys, updated the aws.jceks entry [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /nsswitch.conf s3a://kartik-test.s3-us-west-2.amazonaws.com 17/01/14 20:14:59 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist But I am able to browse the bucket in http ... View more

Re: How to install the hadoop-aws module to copy f...

by Contributor in Support Questions
‎01-15-2017 12:07 AM
‎01-15-2017 12:07 AM
Getting there....I missed a colon in my previous attempt...... [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws.jceks /nsswitch.conf s3a://kartik-test.s3-us-west-2.amazonaws.com 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Caught an AmazonServiceException, which means your request made it to Amazon S3, but was rejected with an error response for some reason. 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Error Message: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 3094C5772AA3B4C0, AWS Error Code: SignatureDoesNotMatch, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method. 17/01/14 15:12:31 INFO s3a.S3AFileSystem: HTTP Status Code: 403 17/01/14 15:12:31 INFO s3a.S3AFileSystem: AWS Error Code: SignatureDoesNotMatch 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Error Type: Client 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Request ID: 3094C5772AA3B4C0 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Class Name: com.amazonaws.services.s3.model.AmazonS3Exception ... View more

Re: How to install the hadoop-aws module to copy f...

by Contributor in Support Questions
‎01-14-2017 09:02 PM
‎01-14-2017 09:02 PM
I tried: hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws.jceks /nsswitch.conf s3a//kartik-test.s3-us-west-2.amazonaws.com and it created a s3a folder in my hdfs: [hdfs@test232 ~]$ hdfs dfs -ls Found 3 items drwx------ - hdfs hdfs 0 2017-01-14 07:47 .Trash drwx------ - hdfs hdfs 0 2017-01-14 12:07 .staging drwx------ - hdfs hdfs 0 2017-01-14 12:07 s3a [hdfs@test232 ~]$ ... View more

Re: How to install the hadoop-aws module to copy f...

by Contributor in Support Questions
‎01-14-2017 07:38 PM
‎01-14-2017 07:38 PM
[hdfs@test232 ~]$ curl http://kartik-test.s3-us-west-2.amazonaws.com <?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>kartik-test</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>hosts</Key><LastModified>2017-01-12T19:48:14.000Z</LastModified><ETag>"881dc3861c3c8a28e213790785a940b7"</ETag><Size>44</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>logs/</Key><LastModified>2017-01-14T17:01:56.000Z</LastModified><ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag><Size>0</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>[hdfs@test232 ~]$ ... View more

Re: How to install the hadoop-aws module to copy f...

by Contributor in Support Questions
‎01-14-2017 04:46 PM
‎01-14-2017 04:46 PM
When I add the access key and the secret in the prompt for the password, I get this: [hdfs@test232 ~]$ hdfs dfs -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks -ls s3a://s3-us-west-2.amazonaws.com/kartik-test 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Caught an AmazonServiceException, which means your request made it to Amazon S3, but was rejected with an error response for some reason. 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Error Message: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: C3EFA25EC200D255, AWS Error Code: null, AWS Error Message: Forbidden 17/01/14 07:51:00 INFO s3a.S3AFileSystem: HTTP Status Code: 403 17/01/14 07:51:00 INFO s3a.S3AFileSystem: AWS Error Code: null 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Error Type: Client 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Request ID: C3EFA25EC200D255 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Class Name: com.amazonaws.services.s3.model.AmazonS3Exception -ls: Fatal internal error ... View more

Re: How to install the hadoop-aws module to copy f...

by Contributor in Support Questions
‎01-14-2017 04:34 PM
‎01-14-2017 04:34 PM
When I run: hadoop credential create fs.s3a.access.key -provider localjceks://file/path/to/aws.jceks <enter AccessKey value at prompt> hadoop credential create fs.s3a.secret.key -provider localjceks://file/path/to/aws.jceks <enter SecretKey value at prompt> It prompts me for a password: [root@test232 conf]# hadoop credential create fs.s3a.access.key -provider localjceks://file/var/tmp/aws.jceks Enter password: Enter password again: ... View more

How to install the hadoop-aws module to copy from ...

by Contributor in Support Questions
‎01-12-2017 08:25 PM
‎01-12-2017 08:25 PM
How to install the hadoop-aws module to copy from on premist hdfs to s3 aws, I need the command s3DistCp ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎06-12-2016 11:30 AM
Community Statistics
Member Since ‎06-12-2016 08:10 AM
Last Visited ‎06-12-2016 11:30 AM
Posts 22
Kudos received 3