About ddolecki

ddolecki · ‎01-15-2017

This worked: [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /test s3a://kartik-test/ Thanks for all your help!!

ddolecki · ‎01-15-2017

This worked: [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /test s3a://kartik-test/ Thanks for all your help!!

ddolecki · ‎01-15-2017

I am using this https://community.hortonworks.com/articles/59161/using-hadoop-credential-api-to-store-aws-secrets.html as a reference, but unable to successfully distcp or even -ls. [root@test232 conf]# hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /nsswitch.conf s3a://kartik-test.s3-us-west-2.amazonaws.com 17/01/15 08:16:22 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:229) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:224) at org.apache.hadoop.tools.DistCp.run(DistCp.java:116) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:454) Invalid arguments: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist I am now reading this: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html but unable to understand what are the xml file names that need to be edited.

ddolecki · ‎01-15-2017

Regenerated keys, updated the aws.jceks entry [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks /nsswitch.conf s3a://kartik-test.s3-us-west-2.amazonaws.com 17/01/14 20:14:59 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Bucket kartik-test.s3-us-west-2.amazonaws.com does not exist But I am able to browse the bucket in http

ddolecki · ‎01-15-2017

Getting there....I missed a colon in my previous attempt...... [hdfs@test232 ~]$ hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws.jceks /nsswitch.conf s3a://kartik-test.s3-us-west-2.amazonaws.com 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Caught an AmazonServiceException, which means your request made it to Amazon S3, but was rejected with an error response for some reason. 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Error Message: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 3094C5772AA3B4C0, AWS Error Code: SignatureDoesNotMatch, AWS Error Message: The request signature we calculated does not match the signature you provided. Check your key and signing method. 17/01/14 15:12:31 INFO s3a.S3AFileSystem: HTTP Status Code: 403 17/01/14 15:12:31 INFO s3a.S3AFileSystem: AWS Error Code: SignatureDoesNotMatch 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Error Type: Client 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Request ID: 3094C5772AA3B4C0 17/01/14 15:12:31 INFO s3a.S3AFileSystem: Class Name: com.amazonaws.services.s3.model.AmazonS3Exception

ddolecki · ‎01-14-2017

I tried: hadoop distcp -Dhadoop.security.credential.provider.path=jceks://hdfs/aws.jceks /nsswitch.conf s3a//kartik-test.s3-us-west-2.amazonaws.com and it created a s3a folder in my hdfs: [hdfs@test232 ~]$ hdfs dfs -ls Found 3 items drwx------ - hdfs hdfs 0 2017-01-14 07:47 .Trash drwx------ - hdfs hdfs 0 2017-01-14 12:07 .staging drwx------ - hdfs hdfs 0 2017-01-14 12:07 s3a [hdfs@test232 ~]$

ddolecki · ‎01-14-2017

[hdfs@test232 ~]$ curl http://kartik-test.s3-us-west-2.amazonaws.com <?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>kartik-test</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>hosts</Key><LastModified>2017-01-12T19:48:14.000Z</LastModified><ETag>"881dc3861c3c8a28e213790785a940b7"</ETag><Size>44</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>logs/</Key><LastModified>2017-01-14T17:01:56.000Z</LastModified><ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag><Size>0</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>[hdfs@test232 ~]$

ddolecki · ‎01-14-2017

When I add the access key and the secret in the prompt for the password, I get this: [hdfs@test232 ~]$ hdfs dfs -Dhadoop.security.credential.provider.path=jceks://hdfs/aws/aws.jceks -ls s3a://s3-us-west-2.amazonaws.com/kartik-test 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Caught an AmazonServiceException, which means your request made it to Amazon S3, but was rejected with an error response for some reason. 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Error Message: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: C3EFA25EC200D255, AWS Error Code: null, AWS Error Message: Forbidden 17/01/14 07:51:00 INFO s3a.S3AFileSystem: HTTP Status Code: 403 17/01/14 07:51:00 INFO s3a.S3AFileSystem: AWS Error Code: null 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Error Type: Client 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Request ID: C3EFA25EC200D255 17/01/14 07:51:00 INFO s3a.S3AFileSystem: Class Name: com.amazonaws.services.s3.model.AmazonS3Exception -ls: Fatal internal error

ddolecki · ‎01-14-2017

When I run: hadoop credential create fs.s3a.access.key -provider localjceks://file/path/to/aws.jceks <enter AccessKey value at prompt> hadoop credential create fs.s3a.secret.key -provider localjceks://file/path/to/aws.jceks <enter SecretKey value at prompt> It prompts me for a password: [root@test232 conf]# hadoop credential create fs.s3a.access.key -provider localjceks://file/var/tmp/aws.jceks Enter password: Enter password again:

ddolecki · ‎01-12-2017

How to install the hadoop-aws module to copy from on premist hdfs to s3 aws, I need the command s3DistCp

ddolecki · ‎07-25-2016

That answered my question. Thank you very much.

ddolecki · ‎07-25-2016

Thanks team. My question has more to do with the scenario in which replication is not possible on a particular node of the cluster because that disk has filled up.

ddolecki · ‎07-25-2016

Thanks for helping me, the information you provided is not unhelpful. I however need to know in a cluster of assorted/different disk sizes what is the limiting factor, is it the size of the smallest disk?

ddolecki · ‎07-25-2016

Let's assume I have a 10 node Hadoop cluster with 1 namenode and 9 datanodes(each with different sized disks). What is the limiting factor to the size of the HDFS filesystem, is it limited to the size of the smallest disk on any of the 9 nodes? I am assuming that this should be the case for a default replication factor of three.

ddolecki · ‎07-15-2016

for the namenode and for datanodes and for yarn/spark? Or is the default provided by Ambari suitable for production use?

ddolecki · ‎06-15-2016

Hi, I'd like to know if downloading the sandbox and practising on it is necessary and sufficient for hadoop administration certification, or if there is another download available which is more geared towards certification. Please advise. BR

ddolecki · ‎06-12-2016

How was this fixed?

Online	Offline
Last Visited	‎06-12-2016 11:30 AM

Date Registered	‎06-12-2016 08:10 AM
Last Visited	‎06-12-2016 11:30 AM
Total Messages Posted	22
Total Kudos Received	3

Re: Unable to distcp from on premise hdfs to s3

Re: Unable to distcp from on premise hdfs to s3

Re: How to install the hadoop-aws module to copy f...

Unable to distcp from on premise hdfs to s3

Re: How to install the hadoop-aws module to copy f...

Re: How to install the hadoop-aws module to copy f...

Re: How to install the hadoop-aws module to copy f...

Re: How to install the hadoop-aws module to copy f...

Re: How to install the hadoop-aws module to copy f...

Re: How to install the hadoop-aws module to copy f...

How to install the hadoop-aws module to copy from ...

Re: Size of HDFS

Re: Size of HDFS

Re: Size of HDFS

Size of HDFS

Do I need to tune Java heap size

What is the best download to prepare for hadoop ad...

Re: Issue on running spark application in Yarn-clu...