Cloudera Community
leonidzav
user rank
Explorer
My Accepted Solutions
Show All

Re: Running distcp between two cluster: One Kerber...

by Explorer in Support Questions
‎03-07-2017 08:28 PM
‎03-07-2017 08:28 PM
We had similar issue. Most likely you have following property in /etc/hadoop/conf/hdfs-site.xml: <property> <name>dfs.namenode.acls.enabled</name> <value>true</value> </property Remove this property or set it to "false". It should help. ... View more

Re: access amazon S3 bucket from hdfs

by Explorer in Support Questions
‎07-16-2016 01:20 AM
1 Kudo
‎07-16-2016 01:20 AM
1 Kudo
Hi All, Eventually I've found the way how to specify Keys on command line: hadoop fs -Dfs.s3a.access.key=<AccessKeyId> -Dfs.s3a.secret.key=<SecurityAccessKey> -Dfs.s3a.proxy.host=<proxy_host> -Dfs.s3a.proxy.port=<proxy_port> -ls s3a://<my_bucket/ Thanks to Constantin and kvarakantham for their responses. . ... View more

Re: access amazon S3 bucket from hdfs

by Explorer in Support Questions
‎07-16-2016 12:58 AM
‎07-16-2016 12:58 AM
Thank you Constantin, Yes, putting Keys on the command line is not very secure. I am agree with you. But if I will set AccessKeyId and SecureAccessKey in core-site.xml, than all hadoop users will be able to access amazon s3 bucket from hadoop. I am trying to avoid this scenario. I am playing with putting Keys on command line, but still not successful with it... Not sure what cause of error - syntax seems OK (now I am trying s3a instead of s3n). ... View more

access amazon S3 bucket from hdfs

by Explorer in Support Questions
‎07-15-2016 08:52 PM
‎07-15-2016 08:52 PM
I am trying to connect amazon S3 bucket from hdfs using this command: $ hadoop fs -ls s3n://<ACCESSKEYID>:<SecretAccessKey>@<bucket-name>/tpt_files/ -ls: Invalid hostname in URI s3n://<ACCESSKEYID>:<SecretAccessKey>@<bucket-name>/tpt_files Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [<path> ...] My SecretAccessKey includes “/”. Could it be cause of such behavior? In the same time I have aws cli installed in this server and I can access by bucket using aws without any issues using command (AccessKeyId and SecretAccessKey configured in .aws/credentials) aws s3 ls s3:// <bucket-name>/tpt_files/ If there any way how to access amazon S3 bucket using Hadoop command without specifying Keys in core-site.xml. I’d prefer to specify Keys in command line. Any suggestions will be very helpful. ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎03-07-2017 08:28 PM
Community Statistics
Member Since ‎01-13-2016 04:45 PM
Last Visited ‎03-07-2017 08:28 PM
Posts 6
Kudos received 2