Created 01-04-2021 07:39 AM
Hello Cloudera community,
Could you please help me about Yarn issue below ? I'm new in Hadoop and I don't know very well hadoop cluster.
21/01/04 11:21:41 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm16
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528130, maxDate=1608841328130, sequenceNumber=288, masterKeyId=735) can't be found in cache
21/01/04 11:21:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528130, maxDate=1608841328130, sequenceNumber=288, masterKeyId=735) can't be found in cache
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528130, maxDate=1608841328130, sequenceNumber=288, masterKeyId=735) can't be found in cache
Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.
21/01/04 11:21:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/01/04 11:21:41 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
21/01/04 11:21:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
21/01/04 11:21:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
at org.apache.hadoop.ipc.Client.call(Client.java:1504)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy15.getListing(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:588)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy16.getListing(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2145)
at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:211)
at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:198)
at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:180)
at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:180)
at org.apache.hadoop.fs.FileContext$21.next(FileContext.java:1468)
at org.apache.hadoop.fs.FileContext$21.next(FileContext.java:1463)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1463)
at org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAllContainersLogs(LogCLIHelpers.java:156)
at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:133)
at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:186)
Thank you very much in advance.
Created 01-04-2021 12:55 PM
This community helps in 2 of the most used Hadoop flavors Cloudera and Hortonworks and these 2 software vendors handled and configured differently their Kerberos.
In cloudera the keytabs are found in /run/cloudera-scm-agent/process/* while in hortonworks it's in /etc/security/keytabs/* so it would be good if you clearly stated.
Please include the description of your cluster too like HA or not I see from the log failover to rm16 which suggest you have RM HA?
Has the user kinited before attempting the operation. Is user impersonating cabhbwg
Happy hadooping