Member since
01-13-2016
6
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 8472 | 07-16-2016 01:20 AM |
03-07-2017
08:28 PM
We had similar issue. Most likely you have following property in /etc/hadoop/conf/hdfs-site.xml: <property>
<name>dfs.namenode.acls.enabled</name>
<value>true</value>
</property Remove this property or set it to "false". It should help.
... View more
07-16-2016
01:20 AM
1 Kudo
Hi All, Eventually I've found the way how to specify Keys on command line: hadoop fs -Dfs.s3a.access.key=<AccessKeyId> -Dfs.s3a.secret.key=<SecurityAccessKey> -Dfs.s3a.proxy.host=<proxy_host> -Dfs.s3a.proxy.port=<proxy_port> -ls s3a://<my_bucket/ Thanks to Constantin and kvarakantham for their responses. .
... View more
07-16-2016
12:58 AM
Thank you Constantin, Yes, putting Keys on the command line is not very secure. I am agree with you. But if I will set AccessKeyId and SecureAccessKey in core-site.xml, than all hadoop users will be able to access amazon s3 bucket from hadoop. I am trying to avoid this scenario. I am playing with putting Keys on command line, but still not successful with it... Not sure what cause of error - syntax seems OK (now I am trying s3a instead of s3n).
... View more
07-15-2016
08:52 PM
I am trying to connect amazon S3 bucket from hdfs using this command: $ hadoop fs -ls s3n://<ACCESSKEYID>:<SecretAccessKey>@<bucket-name>/tpt_files/ -ls: Invalid hostname in URI s3n://<ACCESSKEYID>:<SecretAccessKey>@<bucket-name>/tpt_files Usage: hadoop fs [generic options] -ls [-d] [-h] [-R]
[<path> ...] My SecretAccessKey includes “/”. Could it be cause of such
behavior? In the same time I have aws cli installed in this server and
I can access by bucket using aws without any issues using command (AccessKeyId
and SecretAccessKey configured in .aws/credentials) aws s3 ls s3:// <bucket-name>/tpt_files/ If there any way how to access amazon S3 bucket using Hadoop
command without specifying Keys in core-site.xml. I’d prefer to specify Keys in
command line. Any suggestions will be very helpful.
... View more
Labels:
07-15-2016
07:47 PM
I am trying to copy data
from amazon S3 bucket to HDFS using distcp command. It failed with following error:
$ hadoop distcp s3n://my_bucket/Sample/2016-03-31.txt
hdfs://Fs.defaultFS/user/test/tmp
16/07/15 11:09:30 INFO tools.DistCp: Input
Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[s3n://my_bucket/Sample/2016-03-31.txt],
targetPath=hdfs://Fs.defaultFS/user/test/tmp, targetPathExists=true,
preserveRawXattrs=false}
16/07/15 11:09:30 INFO
impl.TimelineClientImpl: Timeline service address: http://my_host.com:8188/ws/v1/timeline/
16/07/15 11:09:31 INFO utils.RestUtils: Using
Proxy: XXX.XXX.XXX.XX:80
16/07/15 11:09:31 INFO utils.RestUtils: Using
Proxy: XXX.XXX.XXX.XX:80
16/07/15 11:09:32 INFO
impl.TimelineClientImpl: Timeline service address: http://my_host.com:8188/ws/v1/timeline/
16/07/15 11:09:32 INFO mapreduce.JobSubmitter:
number of splits:1
16/07/15 11:09:33 INFO mapreduce.JobSubmitter:
Submitting tokens for job: job_1468432612016_3625
16/07/15 11:09:33 INFO impl.YarnClientImpl:
Submitted application application_1468432612016_3625
16/07/15 11:09:33 INFO mapreduce.Job: The url
to track the job: http://my_host.com:8088/proxy/application_1468432612016_3625/
16/07/15 11:09:33 INFO tools.DistCp: DistCp
job-id: job_1468432612016_3625
16/07/15 11:09:33 INFO mapreduce.Job: Running
job: job_1468432612016_3625
16/07/15 11:09:39 INFO mapreduce.Job: Job
job_1468432612016_3625 running in uber mode : false
16/07/15 11:09:39 INFO mapreduce.Job: map 0% reduce 0%
16/07/15 11:09:43 INFO mapreduce.Job: Task Id
: attempt_1468432612016_3625_m_000000_0, Status : FAILED
Error: java.net.UnknownHostException: my_bucket.s3.amazonaws.com
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
at java.net.InetAddress.getAllByName0(InetAddress.java:1246)
at java.net.InetAddress.getAllByName(InetAddress.java:1162)
at java.net.InetAddress.getAllByName(InetAddress.java:1098)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:278)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:162)
at
org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151)
at
org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:334)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:942)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2148)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2075)
at
org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1093)
at
org.jets3t.service.StorageService.getObjectDetails(StorageService.java:548)
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at org.apache.hadoop.fs.s3native.$Proxy17.retrieveMetadata(Unknown
Source)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:476)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
16/07/15 11:09:48 INFO mapreduce.Job: Task Id
: attempt_1468432612016_3625_m_000000_1, Status : FAILED
Error: java.net.UnknownHostException: my_bucket.s3.amazonaws.com
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
at java.net.InetAddress.getAllByName0(InetAddress.java:1246)
at java.net.InetAddress.getAllByName(InetAddress.java:1162)
at java.net.InetAddress.getAllByName(InetAddress.java:1098)
at
org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:278)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:162)
at
org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151)
at
org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:334)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:942)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2148)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2075)
at
org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1093)
at
org.jets3t.service.StorageService.getObjectDetails(StorageService.java:548)
at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at org.apache.hadoop.fs.s3native.$Proxy17.retrieveMetadata(Unknown
Source)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:476)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
16/07/15 11:09:53 INFO mapreduce.Job: Task Id
: attempt_1468432612016_3625_m_000000_2, Status : FAILED
Error: java.net.UnknownHostException: my_bucket.s3.amazonaws.com
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
at java.net.InetAddress.getAllByName0(InetAddress.java:1246)
at java.net.InetAddress.getAllByName(InetAddress.java:1162)
at java.net.InetAddress.getAllByName(InetAddress.java:1098)
at
org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:278)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:162)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151)
at
org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:334)
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:942)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2148)
at
org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2075)
at
org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1093)
at
org.jets3t.service.StorageService.getObjectDetails(StorageService.java:548)
at
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at org.apache.hadoop.fs.s3native.$Proxy17.retrieveMetadata(Unknown
Source)
at
org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:476)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
16/07/15 11:09:59 INFO mapreduce.Job: map 100% reduce 0%
16/07/15 11:09:59 INFO mapreduce.Job: Job
job_1468432612016_3625 failed with state FAILED due to: Task failed
task_1468432612016_3625_m_000000
Job failed as tasks failed. failedMaps:1
failedReduces:0
16/07/15 11:09:59 INFO mapreduce.Job:
Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=13485
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13485
Total vcore-seconds taken by all map tasks=26970
Total megabyte-seconds taken by all map tasks=13808640
16/07/15 11:09:59 ERROR tools.DistCp:
Exception encountered
java.io.IOException: DistCp failure: Job
job_1468432612016_3625 has failed: Task failed task_1468432612016_3625_m_000000
Job failed as tasks failed. failedMaps:1
failedReduces:0
at org.apache.hadoop.tools.DistCp.waitForJobCompletion(DistCp.java:205)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:156)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:126)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
Here is some of my config
parameters: HDP version 2.4.0 Following properties
were added to core-site.xml: <property>
<name>fs.s3n.awsAccessKeyId</name>
<value>XXXXXXXXXXXXXXXXX</value>
</property>
<property>
<name>fs.s3n.awsSecretAccessKey</name>
<value>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</value>
</property>
File jets3t.properties
was created in /etc/hadoop/conf: $ cat jets3t.properties httpclient.proxy-autodetect=false httpclient.proxy-host=XXX.XXX.XXX.XX httpclient.proxy-port=80 I am able to run Hadoop
fs –ls , Hadoop fs –cp and so on. I have issue only with distcp command. If anyone know how this
issue can be resolved?
... View more
Labels:
02-09-2016
03:29 PM
1 Kudo
I am trying to install hortonworks cluster for 10 nodes
with local repositories using Ambari. OS - Red Hat Enterprise Linux Server release 6.4
(Santiago). HDP 2.3.4.0-3485, HDP-UTILS-1.1.0.20, Ambari 2.2.0.0-1310. On step 12 (Install, Start and Test) installation failed,
unable to install HDFS Client. Here is the full error message: stderr: /var/lib/ambari-agent/data/errors-47.txt
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 120, in <module>
HdfsClient().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 36, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 41, in configure
hdfs()
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs.py", line 61, in hdfs
group=params.user_group
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/xml_config.py", line 67, in action_create
encoding = self.resource.encoding
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 87, in action_create
raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))
resource_management.core.exceptions.Fail: Applying File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] failed, parent directory /usr/hdp/current/hadoop-client/conf doesn't exist In directory /usr/hdp
exist only subdirectory “current”. No subdirectory “2.3.4.0-3485” Before this installation
I had another cluster on those nodes – phd. I removed all repos from /etc/yum.repos.d/
and remove all links to PHD repositories from /var/www/html. But during installation
subfolders created under the /usr/phd. In directory /etc/yum.repos.d/ no phd.repo # ls -l /etc/yum.repos.d/
total 84
-rw-r--r--. 1 root root456 Feb9 00:59 ambari.repo
-rw-r--r--. 1 root root1059 Dec 19 02:07 epel.repo
-rw-r--r--. 1 root root940 Feb9 00:54 hdp.repo
-rw-r--r--. 1 root root125 Feb9 01:08 HDP.repo
-rw-r--r--. 1 root root148 Feb9 01:08 HDP-UTILS.repo
-rw-r--r--. 1 root root 58473 Feb9 09:56 redhat.repo
-rw-r--r--. 1 root root529 Dec 18 14:58 rhel-source.repo
Yum repolist don’t show
any other repositories # yum repolist
Loaded plugins:
priorities, product-id, security, subscription-manager
This system is
receiving updates from Red Hat Subscription Management.
Repository
HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Trying other mirror.
rhel-6-server-rpms| 3.7 kB00:00
rhel-server-dts-6-rpms| 2.9 kB00:00
rhel-server-dts2-6-rpms| 2.9 kB00:00
175 packages excluded
due to repository priority protections
repo idrepo namestatus
HDP-2.3HDP-2.30+175
HDP-2.3.4.0HDP Version - HDP-2.3.4.0175
HDP-UTILS-1.1.0.20HDP-UTILS-1.1.0.2043
Updates-ambari-2.2.0.0ambari-2.2.0.0 - Updates7
epelExtra Packages for
Enterprise Linux 6 - x86_6411,992
rhel-6-server-rpmsRed Hat Enterprise Linux 6 Server
(RPMs)16,510
rhel-server-dts-6-rpmsRed Hat Developer Toolset RPMs for Red Hat
Enterprise Linux 6 Serv84
rhel-server-dts2-6-rpms
Red Hat Developer Toolset 2 RPMs for Red Hat Enterprise Linux 6 Se469
repolist: 29,280
What settings do I need
to install hortonworks cluster in this case?
... View more
