Member since
10-20-2016
106
Posts
0
Kudos Received
0
Solutions
10-21-2019
12:59 AM
Please find the O/P <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="REFRESH" content="0;url=dfshealth.html" /> <title>Hadoop Administration</title> </head> </html>
... View more
10-20-2019
11:28 PM
@paras Now able to access the namenode UI after disabling kerbeos. But, I see some error in UI Please find the screenshot
... View more
10-18-2019
02:09 AM
@paras I have disabled the kerberos, but still the name node UI is not working [root@hostname~]# netstat -an | grep 50070 tcp 0 0 10.49.70.13:50070 0.0.0.0:* LISTEN tcp 0 0 10.49.70.13:50070 10.49.70.13:41944 TIME_WAIT tcp 0 0 10.49.70.13:50070 10.49.70.13:41904 TIME_WAIT tcp 0 0 10.49.70.13:50070 10.49.70.13:42070 ESTABLISHED tcp 0 0 10.49.70.13:50070 10.49.70.13:41902 TIME_WAIT tcp 0 0 10.49.70.13:50070 10.49.70.13:41898 TIME_WAIT tcp 0 0 10.49.70.13:50070 10.49.70.13:41908 TIME_WAIT tcp 0 0 10.49.70.13:50070 10.49.70.13:41900 TIME_WAIT tcp 0 0 10.49.70.13:50070 10.49.70.13:42064 ESTABLISHED tcp 0 0 10.49.70.13:50070 10.49.70.14:58658 TIME_WAIT tcp 0 0 10.49.70.13:50070 10.49.70.13:41906 TIME_WAIT tcp6 0 0 10.49.70.13:41876 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:46082 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:45984 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:42064 10.49.70.13:50070 ESTABLISHED tcp6 0 0 10.49.70.13:45912 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:45880 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:41760 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:46170 10.49.70.14:50070 ESTABLISHED tcp6 0 0 10.49.70.13:45946 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:41798 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:45838 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:41976 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:41842 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:45914 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:41882 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:41764 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:46144 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:41840 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:45988 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:41978 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:46084 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:46140 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:45836 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:42070 10.49.70.13:50070 ESTABLISHED tcp6 0 0 10.49.70.13:46172 10.49.70.14:50070 ESTABLISHED tcp6 0 0 10.49.70.13:41794 10.49.70.13:50070 TIME_WAIT tcp6 0 0 10.49.70.13:45876 10.49.70.14:50070 TIME_WAIT tcp6 0 0 10.49.70.13:42024 10.49.70.13:50070 TIME_WAIT
... View more
10-17-2019
06:33 AM
it is applicable for HWX also? because , I have added the below property in core-site-xml while enabling hadoop.http.authentication.simple.anonymous.allowed false hadoop.http.authentication.signature.secret.file /etc/security/http_secret hadoop.http.authentication.type kerberos hadoop.http.authentication.kerberos.keytab /etc/security/keytabs/spnego.service.keytab hadoop.http.authentication.kerberos.principal HTTP/_HOST@ EXAMPLE.COM hadoop.http.filter.initializers org.apache.hadoop.security.AuthenticationFilterInitializer hadoop.http.authentication.cookie.domain hortonworks.local
... View more
10-17-2019
05:25 AM
it is possible to disable SPNEGO autheneticaion with kerberos enabled cluster.
... View more
10-17-2019
03:44 AM
windows 10 here
... View more
10-17-2019
03:31 AM
ksetup has done and changed the below property in mozilla firefoz as per HWX docs, but still the name node UI is not opening For FireFox: Navigate to the about:config URL (type about:config in the address box, then press the Enter key). Scroll down to network.negotiate-auth.trusted-uris and change its value to your cluster domain name (For example, .hwx.site). Change the value of network.negotiate-auth.delegation-uris to your cluster domain name (For example, .hwx.site).
... View more
10-17-2019
02:46 AM
ok. ksetup /addhosttorealmmap <httpFS hostname> <REALM> httpFS hostname mean namenode host name ?
... View more
10-17-2019
01:19 AM
I am getting below error while doing ktsetup Failed to create Kerberos key: 5 (0x5) Failed to open Kerberos Key: 0x5 Failed /AddKdc : 0xc0000001
... View more
10-17-2019
01:11 AM
the variables are configured in the /etc/krb5.conf
... View more
10-17-2019
12:53 AM
I have two realms in my environment.
for ex. forest.org - currently all users are mapped to this domain ( Redhat IDM)
data.forest.org - Local MIT KDC (hadoop principlas)
Currently user@forest.org is facing issues while accessing hdfs. Please find the error
19/10/17 03:43:38 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1571298216320
Kindly help us to fix this issue.
... View more
Labels:
- Labels:
-
Apache Hadoop
10-16-2019
06:11 AM
After enabling kerberos, unable to access any of the Web UI. As per HWX docs, SPNEGO has been enabled, but still facing issue in accessing the Web UI.
... View more
Labels:
- Labels:
-
Apache Hadoop
10-16-2019
02:57 AM
@Shelton After changing the realm in KDC.conf, now able to execute HDFS commands. After enabling kerberos, unable to access the any of the WEB UI like hdfs,yarn,mapreduce
... View more
10-15-2019
10:13 AM
@Shelton not able to execute any of the HDFS command due to kerberos. hadoop fs -ls / 19/10/15 13:12:55 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/10/15 13:12:55 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/10/15 13:12:55 INFO retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort hostname:8020 , LocalHost:localPort hostname/10.49.70.18:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over hostname10.49.70.14:8020 after 1 failover attempts. Trying to failover after sleeping for 1171ms
... View more
10-14-2019
11:27 PM
@Shelton big file transfer asking mail id. can you please share it, so that i can send it
... View more
10-14-2019
07:48 PM
@Shelton I am unable to add the kadmin and krbkdc logs in the bods, since it contains more lines. any idea how to attach the logs
... View more
10-14-2019
07:40 PM
@Shelton yes shelton, installed all the keberos libs and packages in all the cluster hosts. After enabling kerberos, I am trying to access hdfs from the client. But, after enabling kerberos it is not allowing me to access namenode, also namenode lost it s high availability. Attaching the Krbkdc and kadmin logs for your reference. Kindly do the needful.
... View more
10-14-2019
06:12 AM
Please find the logs.
$ HADOOP_ROOT_LOGGER=DEBUG,console hdfs dfs -ls / 19/10/14 08:59:25 DEBUG util.Shell: setsid exited with exit code 0 19/10/14 08:59:25 DEBUG conf.Configuration: parsing URL jar:file:/usr/hdp/3.0.1.0-187/hadoop/hadoop-common-3.1.1.3.0.1.0-187.jar!/core-default.xml 19/10/14 08:59:25 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@66480dd7 19/10/14 08:59:25 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/3.0.1.0-187/0/core-site.xml 19/10/14 08:59:25 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@1877ab81 19/10/14 08:59:25 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true 19/10/14 08:59:25 DEBUG security.Groups: Creating new Groups object 19/10/14 08:59:25 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... 19/10/14 08:59:25 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library 19/10/14 08:59:25 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution 19/10/14 08:59:25 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping 19/10/14 08:59:25 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000 19/10/14 08:59:25 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 19/10/14 08:59:25 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 19/10/14 08:59:25 DEBUG security.UserGroupInformation: hadoop login 19/10/14 08:59:25 DEBUG security.UserGroupInformation: hadoop login commit 19/10/14 08:59:25 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: hdfs 19/10/14 08:59:25 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: hdfs" with name hdfs 19/10/14 08:59:25 DEBUG security.UserGroupInformation: User entry: "hdfs" 19/10/14 08:59:25 DEBUG security.UserGroupInformation: UGI loginUser:hdfs (auth:SIMPLE) 19/10/14 08:59:25 DEBUG core.Tracer: sampler.classes = ; loaded no samplers 19/10/14 08:59:25 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers 19/10/14 08:59:25 DEBUG fs.FileSystem: Loading filesystems 19/10/14 08:59:25 DEBUG fs.FileSystem: file:// = class org.apache.hadoop.fs.LocalFileSystem from /usr/hdp/3.0.1.0-187/hadoop/hadoop-common-3.1.1.3.0.1.0-187.jar 19/10/14 08:59:25 DEBUG fs.FileSystem: viewfs:// = class org.apache.hadoop.fs.viewfs.ViewFileSystem from /usr/hdp/3.0.1.0-187/hadoop/hadoop-common-3.1.1.3.0.1.0-187.jar 19/10/14 08:59:25 DEBUG fs.FileSystem: har:// = class org.apache.hadoop.fs.HarFileSystem from /usr/hdp/3.0.1.0-187/hadoop/hadoop-common-3.1.1.3.0.1.0-187.jar 19/10/14 08:59:25 DEBUG fs.FileSystem: http:// = class org.apache.hadoop.fs.http.HttpFileSystem from /usr/hdp/3.0.1.0-187/hadoop/hadoop-common-3.1.1.3.0.1.0-187.jar 19/10/14 08:59:25 DEBUG fs.FileSystem: https:// = class org.apache.hadoop.fs.http.HttpsFileSystem from /usr/hdp/3.0.1.0-187/hadoop/hadoop-common-3.1.1.3.0.1.0-187.jar 19/10/14 08:59:25 DEBUG fs.FileSystem: hdfs:// = class org.apache.hadoop.hdfs.DistributedFileSystem from /usr/hdp/3.0.1.0-187/hadoop-hdfs/hadoop-hdfs-client-3.1.1.3.0.1.0-187.jar 19/10/14 08:59:25 DEBUG fs.FileSystem: webhdfs:// = class org.apache.hadoop.hdfs.web.WebHdfsFileSystem from /usr/hdp/3.0.1.0-187/hadoop-hdfs/hadoop-hdfs-client-3.1.1.3.0.1.0-187.jar 19/10/14 08:59:25 DEBUG fs.FileSystem: swebhdfs:// = class org.apache.hadoop.hdfs.web.SWebHdfsFileSystem from /usr/hdp/3.0.1.0-187/hadoop-hdfs/hadoop-hdfs-client-3.1.1.3.0.1.0-187.jar 19/10/14 08:59:25 DEBUG gcs.GoogleHadoopFileSystemBase: GHFS version: 1.9.0.3.0.1.0-187 19/10/14 08:59:25 DEBUG fs.FileSystem: gs:// = class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem from /usr/hdp/3.0.1.0-187/hadoop-mapreduce/gcs-connector-1.9.0.3.0.1.0-187-shaded.jar 19/10/14 08:59:25 DEBUG fs.FileSystem: s3n:// = class org.apache.hadoop.fs.s3native.NativeS3FileSystem from /usr/hdp/3.0.1.0-187/hadoop-mapreduce/hadoop-aws-3.1.1.3.0.1.0-187.jar 19/10/14 08:59:25 DEBUG fs.FileSystem: Looking for FS supporting hdfs 19/10/14 08:59:25 DEBUG fs.FileSystem: looking for configuration option fs.hdfs.impl 19/10/14 08:59:26 DEBUG fs.FileSystem: Looking in service filesystems for implementation class 19/10/14 08:59:26 DEBUG fs.FileSystem: FS for hdfs is class org.apache.hadoop.hdfs.DistributedFileSystem 19/10/14 08:59:26 DEBUG impl.DfsClientConf: dfs.client.use.legacy.blockreader.local = false 19/10/14 08:59:26 DEBUG impl.DfsClientConf: dfs.client.read.shortcircuit = true 19/10/14 08:59:26 DEBUG impl.DfsClientConf: dfs.client.domain.socket.data.traffic = false 19/10/14 08:59:26 DEBUG impl.DfsClientConf: dfs.domain.socket.path = /var/lib/hadoop-hdfs/dn_socket 19/10/14 08:59:26 DEBUG hdfs.DFSClient: Sets dfs.client.block.write.replace-datanode-on-failure.min-replication to 0 19/10/14 08:59:26 DEBUG hdfs.HAUtilClient: No HA service delegation token found for logical URI hdfs://datalakeqa 19/10/14 08:59:26 DEBUG impl.DfsClientConf: dfs.client.use.legacy.blockreader.local = false 19/10/14 08:59:26 DEBUG impl.DfsClientConf: dfs.client.read.shortcircuit = true 19/10/14 08:59:26 DEBUG impl.DfsClientConf: dfs.client.domain.socket.data.traffic = false 19/10/14 08:59:26 DEBUG impl.DfsClientConf: dfs.domain.socket.path = /var/lib/hadoop-hdfs/dn_socket 19/10/14 08:59:26 DEBUG retry.RetryUtils: multipleLinearRandomRetry = null 19/10/14 08:59:26 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine$RpcProtobufRequest, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@6babf3bf 19/10/14 08:59:26 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@3d6f0054 19/10/14 08:59:26 DEBUG unix.DomainSocketWatcher: org.apache.hadoop.net.unix.DomainSocketWatcher$2@6129020f: starting with interruptCheckPeriodMs = 60000 19/10/14 08:59:26 DEBUG shortcircuit.DomainSocketFactory: The short-circuit local reads feature is enabled. 19/10/14 08:59:26 DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol using SaslPropertiesResolver, configured QOP dfs.data.transfer.protection = authentication,privacy, configured class dfs.data.transfer.saslproperties.resolver.class = class org.apache.hadoop.security.SaslPropertiesResolver 19/10/14 08:59:26 DEBUG ipc.Client: The ping interval is 60000 ms. 19/10/14 08:59:26 DEBUG ipc.Client: Connecting to /10.49.70.13:8020 19/10/14 08:59:26 DEBUG security.UserGroupInformation: PrivilegedAction as:hdfs (auth:SIMPLE) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:796) 19/10/14 08:59:26 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE
19/10/14 08:59:26 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:@org.apache.hadoop.security.token.TokenInfo(value=class org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector) 19/10/14 08:59:26 DEBUG security.SaslRpcClient: tokens aren't supported for this protocol or user doesn't have one 19/10/14 08:59:26 DEBUG security.SaslRpcClient: client isn't using kerberos 19/10/14 08:59:26 DEBUG security.UserGroupInformation: PrivilegedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/10/14 08:59:26 DEBUG security.UserGroupInformation: PrivilegedAction as:hdfs (auth:SIMPLE) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720) 19/10/14 08:59:26 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/10/14 08:59:26 DEBUG security.UserGroupInformation: PrivilegedActionException as:hdfs (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] 19/10/14 08:59:26 DEBUG ipc.Client: closing ipc connection to /10.49.70.13:8020: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:757) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:720) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:813) at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1558) at org.apache.hadoop.ipc.Client.call(Client.java:1389) at org.apache.hadoop.ipc.Client.call(Client.java:1353) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:65) at org.apache.hadoop.fs.Globber.doGlob(Globber.java:283) at org.apache.hadoop.fs.Globber.glob(Globber.java:149) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2067) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:353) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104) at org.apache.hadoop.fs.shell.Command.run(Command.java:177) at org.apache.hadoop.fs.FsShell.run(FsShell.java:328) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:391) Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:614) at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:410) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:800) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:796) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:796)
... View more
Labels:
10-03-2019
10:42 PM
Every time when restarting yarn node manager out file will be created with read permission. please find the below logs.
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/nodemanager.py", line 102, in <module> Nodemanager().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 351, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/nodemanager.py", line 53, in start service('nodemanager',action='start') File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/service.py", line 93, in service Execute(daemon_cmd, user = usr, not_if = check_process) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run returns=self.resource.returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of 'ulimit -c unlimited; export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.1.0-187/hadoop/libexec && /usr/hdp/3.0.1.0-187/hadoop-yarn/bin/yarn --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon start nodemanager' returned 1. /usr/hdp/3.0.1.0-187/hadoop/libexec/hadoop-functions.sh: line 1847: /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid: Permission denied ERROR: Cannot write nodemanager pid /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid. /usr/hdp/3.0.1.0-187/hadoop/libexec/hadoop-functions.sh: line 1866: /var/log/hadoop-yarn/yarn/hadoop-yarn-nodemanager-.org.out: Permission denied
... View more
Labels:
- Labels:
-
Apache YARN
10-02-2019
11:02 PM
Please find the logs 2019-10-02 13:41:56,043 - Using hadoop conf dir: /usr/hdp/3.0.1.0-187/hadoop/conf 2019-10-02 13:41:56,044 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=3.0.1.0-187 -> 3.0.1.0-187 2019-10-02 13:41:56,081 - Using hadoop conf dir: /usr/hdp/3.0.1.0-187/hadoop/conf 2019-10-02 13:41:56,100 - Execute['export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.1.0-187/hadoop/libexec && /usr/hdp/3.0.1.0-187/hadoop-yarn/bin/yarn --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon stop nodemanager'] {'only_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid', 'user': 'yarn'} 2019-10-02 13:41:56,128 - Skipping Execute['export HADOOP_LIBEXEC_DIR=/usr/hdp/3.0.1.0-187/hadoop/libexec && /usr/hdp/3.0.1.0-187/hadoop-yarn/bin/yarn --config /usr/hdp/3.0.1.0-187/hadoop/conf --daemon stop nodemanager'] due to only_if 2019-10-02 13:41:56,129 - Waiting for actual component stop 2019-10-02 13:42:06,219 - Waiting for actual component stop 2019-10-02 13:42:16,343 - Waiting for actual component stop 2019-10-02 13:42:26,465 - Waiting for actual component stop 2019-10-02 13:42:36,610 - Waiting for actual component stop 2019-10-02 13:42:46,725 - Waiting for actual component stop 2019-10-02 13:42:56,849 - Waiting for actual component stop 2019-10-02 13:43:06,954 - Waiting for actual component stop 2019-10-02 13:43:17,080 - Waiting for actual component stop 2019-10-02 13:43:27,229 - Waiting for actual component stop 2019-10-02 13:43:37,345 - Waiting for actual component stop 2019-10-02 13:43:47,458 - Waiting for actual component stop 2019-10-02 13:43:57,566 - Waiting for actual component stop 2019-10-02 13:44:07,668 - Waiting for actual component stop 2019-10-02 13:44:17,760 - Waiting for actual component stop 2019-10-02 13:44:27,855 - Waiting for actual component stop 2019-10-02 13:44:37,947 - Waiting for actual component stop 2019-10-02 13:44:48,026 - Waiting for actual component stop 2019-10-02 13:44:58,103 - Waiting for actual component stop 2019-10-02 13:45:08,206 - Waiting for actual component stop 2019-10-02 13:45:18,295 - Waiting for actual component stop 2019-10-02 13:45:28,388 - Waiting for actual component stop 2019-10-02 13:45:38,486 - Waiting for actual component stop 2019-10-02 13:45:48,569 - Waiting for actual component stop 2019-10-02 13:45:58,658 - Waiting for actual component stop
... View more
- Tags:
- the logs
10-02-2019
11:01 PM
Labels:
- Labels:
-
Apache YARN
09-30-2019
04:26 AM
Thanks! it is working now
... View more
09-30-2019
04:19 AM
the above host is not able to write the logs in /var/log/ambari-metrics-monitor, don't find the log file for today. total 42M -rw-r--r-- 1 ams hadoop 9.6M Jan 5 2019 ambari-metrics-monitor.log.4 -rw-r--r-- 1 ams hadoop 9.6M Mar 7 2019 ambari-metrics-monitor.log.3 -rw-r--r-- 1 ams hadoop 9.6M May 7 11:34 ambari-metrics-monitor.log.2 -rw-r--r-- 1 ams hadoop 9.6M Aug 2 21:06 ambari-metrics-monitor.log.1 -r--r--r-- 1 ams hadoop 3.1M Aug 22 13:43 ambari-metrics-monitor.log -rw-r--r-- 1 ams hadoop 9.3K Sep 30 06:47 ambari-metrics-monitor.out
... View more
09-30-2019
04:11 AM
Please find the logs Please find the logs HDP version - 3.0 Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_monitor.py", line 78, in <module>
AmsMonitor().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 351, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_monitor.py", line 43, in start
action = 'start'
File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/ams_service.py", line 109, in ams_service
user=params.ams_user
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf start' returned 255. psutil build directory is not empty, continuing...
Verifying Python version compatibility...
Using python /usr/bin/python2.7
Checking for previously running Metric Monitor...
/var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid found with no process. Removing 30543...
Starting ambari-metrics-monitor
Verifying ambari-metrics-monitor process status with PID : 8619
Output of PID check :
ERROR: ambari-metrics-monitor start failed. For more details, see /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
====================
rotateLog = logging.handlers.RotatingFileHandler(config.ams_monitor_log_file(), "a", 10000000, 25)
File "/usr/lib64/python2.7/logging/handlers.py", line 117, in __init__
BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib64/python2.7/logging/handlers.py", line 64, in __init__
logging.FileHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
StreamHandler.__init__(self, self._open())
File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/ambari-metrics-monitor/ambari-metrics-monitor.log'
====================
Monitor out at: /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out stdout : /var/lib/ambari-agent/data/output-5495.txt 2019-09-30 06:47:33,433 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=3.0.1.0-187 -> 3.0.1.0-187
2019-09-30 06:47:33,454 - Using hadoop conf dir: /usr/hdp/3.0.1.0-187/hadoop/conf
2019-09-30 06:47:33,692 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=3.0.1.0-187 -> 3.0.1.0-187
2019-09-30 06:47:33,699 - Using hadoop conf dir: /usr/hdp/3.0.1.0-187/hadoop/conf
2019-09-30 06:47:33,701 - Group['livy'] {}
2019-09-30 06:47:33,703 - Group['spark'] {}
2019-09-30 06:47:33,703 - Group['ranger'] {}
2019-09-30 06:47:33,703 - Group['nifiregistry'] {}
2019-09-30 06:47:33,703 - Group['hdfs'] {}
2019-09-30 06:47:33,704 - Group['hadoop'] {}
2019-09-30 06:47:33,704 - Group['nifi'] {}
2019-09-30 06:47:33,704 - Group['users'] {}
2019-09-30 06:47:33,705 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-09-30 06:47:33,707 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-09-30 06:47:33,708 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-09-30 06:47:33,709 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-09-30 06:47:33,711 - User['superset'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-09-30 06:47:33,712 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-09-30 06:47:33,714 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger', 'hadoop'], 'uid': None}
2019-09-30 06:47:33,715 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-09-30 06:47:33,717 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None}
2019-09-30 06:47:33,718 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None}
2019-09-30 06:47:33,720 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['livy', 'hadoop'], 'uid': None}
2019-09-30 06:47:33,721 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['spark', 'hadoop'], 'uid': None}
2019-09-30 06:47:33,723 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2019-09-30 06:47:33,724 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
2019-09-30 06:47:33,726 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-09-30 06:47:33,727 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-09-30 06:47:33,729 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2019-09-30 06:47:33,730 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2019-09-30 06:47:33,732 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2019-09-30 06:47:33,741 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2019-09-30 06:47:33,742 - Group['hdfs'] {}
2019-09-30 06:47:33,742 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
2019-09-30 06:47:33,743 - FS Type: HDFS
2019-09-30 06:47:33,744 - Directory['/etc/hadoop'] {'mode': 0755}
2019-09-30 06:47:33,760 - File['/usr/hdp/3.0.1.0-187/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-09-30 06:47:33,761 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2019-09-30 06:47:33,784 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2019-09-30 06:47:33,794 - Skipping Execute[('setenforce', '0')] due to not_if
2019-09-30 06:47:33,795 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2019-09-30 06:47:33,798 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2019-09-30 06:47:33,799 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2019-09-30 06:47:33,799 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2019-09-30 06:47:33,806 - File['/usr/hdp/3.0.1.0-187/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2019-09-30 06:47:33,809 - File['/usr/hdp/3.0.1.0-187/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2019-09-30 06:47:33,820 - File['/usr/hdp/3.0.1.0-187/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2019-09-30 06:47:33,842 - File['/usr/hdp/3.0.1.0-187/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2019-09-30 06:47:33,844 - File['/usr/hdp/3.0.1.0-187/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2019-09-30 06:47:33,846 - File['/usr/hdp/3.0.1.0-187/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2019-09-30 06:47:33,854 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2019-09-30 06:47:33,861 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2019-09-30 06:47:33,868 - Skipping unlimited key JCE policy check and setup since it is not required
2019-09-30 06:47:33,879 - Skipping stack-select on AMBARI_METRICS because it does not exist in the stack-select package structure.
2019-09-30 06:47:34,219 - Using hadoop conf dir: /usr/hdp/3.0.1.0-187/hadoop/conf
2019-09-30 06:47:34,223 - checked_call['hostid'] {}
2019-09-30 06:47:34,228 - checked_call returned (0, '310aafc2')
2019-09-30 06:47:34,231 - Directory['/etc/ambari-metrics-monitor/conf'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True}
2019-09-30 06:47:34,233 - Directory['/var/log/ambari-metrics-monitor'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'mode': 0755}
2019-09-30 06:47:34,234 - Execute['ambari-sudo.sh chown -R ams:hadoop /var/log/ambari-metrics-monitor'] {}
2019-09-30 06:47:34,242 - Directory['/var/run/ambari-metrics-monitor'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-09-30 06:47:34,243 - Directory['/usr/lib/python2.6/site-packages/resource_monitoring/psutil/build'] {'owner': 'ams', 'group': 'hadoop', 'create_parents': True, 'cd_access': 'a'}
2019-09-30 06:47:34,244 - Execute['ambari-sudo.sh chown -R ams:hadoop /usr/lib/python2.6/site-packages/resource_monitoring'] {}
2019-09-30 06:47:34,254 - TemplateConfig['/etc/ambari-metrics-monitor/conf/metric_monitor.ini'] {'owner': 'ams', 'template_tag': None, 'group': 'hadoop'}
2019-09-30 06:47:34,263 - File['/etc/ambari-metrics-monitor/conf/metric_monitor.ini'] {'content': Template('metric_monitor.ini.j2'), 'owner': 'ams', 'group': 'hadoop', 'mode': None}
2019-09-30 06:47:34,263 - TemplateConfig['/etc/ambari-metrics-monitor/conf/metric_groups.conf'] {'owner': 'ams', 'template_tag': None, 'group': 'hadoop'}
2019-09-30 06:47:34,265 - File['/etc/ambari-metrics-monitor/conf/metric_groups.conf'] {'content': Template('metric_groups.conf.j2'), 'owner': 'ams', 'group': 'hadoop', 'mode': None}
2019-09-30 06:47:34,271 - File['/etc/ambari-metrics-monitor/conf/ams-env.sh'] {'content': InlineTemplate(...), 'owner': 'ams'}
2019-09-30 06:47:34,277 - Directory['/usr/lib/ambari-logsearch-logfeeder/conf'] {'create_parents': True, 'mode': 0755, 'cd_access': 'a'}
2019-09-30 06:47:34,278 - Generate Log Feeder config file: /usr/lib/ambari-logsearch-logfeeder/conf/input.config-ambari-metrics.json
2019-09-30 06:47:34,278 - File['/usr/lib/ambari-logsearch-logfeeder/conf/input.config-ambari-metrics.json'] {'content': Template('input.config-ambari-metrics.json.j2'), 'mode': 0644}
2019-09-30 06:47:34,280 - Execute['/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf start'] {'user': 'ams'}
2019-09-30 06:47:36,459 - Execute['find /var/log/ambari-metrics-monitor -maxdepth 1 -type f -name '*' -exec echo '==> {} <==' \; -exec tail -n 40 {} \;'] {'logoutput': True, 'ignore_failures': True, 'user': 'ams'}
==> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out <==
StreamHandler.__init__(self, self._open())
File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/ambari-metrics-monitor/ambari-metrics-monitor.log'
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_monitoring/main.py", line 107, in <module>
main()
File "/usr/lib/python2.6/site-packages/resource_monitoring/main.py", line 57, in main
server_process_main(stop_handler)
File "/usr/lib/python2.6/site-packages/resource_monitoring/main.py", line 63, in server_process_main
_init_logging(main_config)
File "/usr/lib/python2.6/site-packages/resource_monitoring/main.py", line 101, in _init_logging
rotateLog = logging.handlers.RotatingFileHandler(config.ams_monitor_log_file(), "a", 10000000, 25)
File "/usr/lib64/python2.7/logging/handlers.py", line 117, in __init__
BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib64/python2.7/logging/handlers.py", line 64, in __init__
logging.FileHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
StreamHandler.__init__(self, self._open())
File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/ambari-metrics-monitor/ambari-metrics-monitor.log'
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_monitoring/main.py", line 107, in <module>
main()
File "/usr/lib/python2.6/site-packages/resource_monitoring/main.py", line 57, in main
server_process_main(stop_handler)
File "/usr/lib/python2.6/site-packages/resource_monitoring/main.py", line 63, in server_process_main
_init_logging(main_config)
File "/usr/lib/python2.6/site-packages/resource_monitoring/main.py", line 101, in _init_logging
rotateLog = logging.handlers.RotatingFileHandler(config.ams_monitor_log_file(), "a", 10000000, 25)
File "/usr/lib64/python2.7/logging/handlers.py", line 117, in __init__
BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib64/python2.7/logging/handlers.py", line 64, in __init__
logging.FileHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
StreamHandler.__init__(self, self._open())
File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/ambari-metrics-monitor/ambari-metrics-monitor.log'
==> /var/log/ambari-metrics-monitor/ambari-metrics-monitor.log.4 <==
2019-01-05 14:19:19,844 [INFO] emitter.py:210 - Calculated collector shard based on hostname : xxxxxxxxx
... View more
09-30-2019
04:10 AM
Labels:
- Labels:
-
Apache Ambari
05-28-2019
11:27 AM
@Geoffrey Shelton Okot please see the above comment
... View more
05-28-2019
11:26 AM
@geoffrey Shelton Okot I have copied the krb5.conf to new host and tried adding the host details in the /etc/hosts. After certain time i see cluster was going down and revoked the /etc/hosts. Please find the log taken from ambari-server
... View more
05-24-2019
09:59 AM
Hi Team, I am unable to add the new host in the existing dev HDP Kerberized cluster. Getting "encountered an error" while installing the HDP. Please find the error in ambari server log:
... View more
Labels:
- Labels:
-
Apache Ambari
05-10-2019
02:49 PM
please find the output [root@edge conf]# /usr/bin/hdp-select | grep hdfs hadoop-hdfs-datanode - 2.4.3.0-227 hadoop-hdfs-journalnode - 2.4.3.0-227 hadoop-hdfs-namenode - 2.4.3.0-227 hadoop-hdfs-nfs3 - 2.4.3.0-227 hadoop-hdfs-portmap - 2.4.3.0-227 hadoop-hdfs-secondarynamenode - 2.4.3.0-227 [root@node1 ~]# /usr/bin/hdp-select | grep hdfs hadoop-hdfs-datanode - 2.4.3.0-227 hadoop-hdfs-journalnode - 2.4.3.0-227 hadoop-hdfs-namenode - 2.4.3.0-227 hadoop-hdfs-nfs3 - 2.4.3.0-227 hadoop-hdfs-portmap - 2.4.3.0-227 hadoop-hdfs-secondarynamenode - 2.4.3.0-227 [root@node1 ~]# [root@edge conf]# yum info hadoop-yarn^C [root@edge conf]# yum info hadoop-hdfs Loaded plugins: fastestmirror, security Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Loading mirror speeds from cached hostfile Available Packages Name : hadoop-hdfs Arch : noarch Version : 2.7.1.2.4.3.0 Release : 227.el6 Size : 2.5 k Repo : HDP-2.4 Summary : hadoop-hdfs Distro virtual package License : APL2 Description : hadoop-hdfs-2.7.1.2.4.3.0 virtual package
... View more
05-10-2019
02:49 PM
HDFS Client has failed in all the three nodes Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 120, in <module> HdfsClient().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 36, in install self.configure(env) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py", line 41, in configure hdfs() File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs.py", line 61, in hdfs group=params.user_group File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/xml_config.py", line 66, in action_create encoding = self.resource.encoding File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 87, in action_create raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname)) resource_management.core.exceptions.Fail: Applying File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] failed, parent directory /usr/hdp/current/hadoop-client/conf doesn't exist
... View more
- « Previous
- Next »
