Support Questions

Find answers, ask questions, and share your expertise

Yarn issue

avatar
New Contributor

Hello Cloudera community, 

 

Could you please help me about Yarn issue below ? I'm new in Hadoop and I don't know very well hadoop cluster.

21/01/04 11:21:41 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm16
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528130, maxDate=1608841328130, sequenceNumber=288, masterKeyId=735) can't be found in cache
21/01/04 11:21:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528130, maxDate=1608841328130, sequenceNumber=288, masterKeyId=735) can't be found in cache
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528130, maxDate=1608841328130, sequenceNumber=288, masterKeyId=735) can't be found in cache
Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.
21/01/04 11:21:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/01/04 11:21:41 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
21/01/04 11:21:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
21/01/04 11:21:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
	at org.apache.hadoop.ipc.Client.call(Client.java:1504)
	at org.apache.hadoop.ipc.Client.call(Client.java:1441)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy15.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:588)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy16.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2145)
	at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:211)
	at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:198)
	at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:180)
	at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:180)
	at org.apache.hadoop.fs.FileContext$21.next(FileContext.java:1468)
	at org.apache.hadoop.fs.FileContext$21.next(FileContext.java:1463)
	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
	at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1463)
	at org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAllContainersLogs(LogCLIHelpers.java:156)
	at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:133)
	at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:186)

 

Thank you very much in advance. 

1 REPLY 1

avatar
Master Mentor

@ibrahima 

This community helps in 2 of the most used Hadoop flavors Cloudera and Hortonworks and these 2 software vendors handled and configured differently their Kerberos.

 

In cloudera the keytabs are found in /run/cloudera-scm-agent/process/* while in hortonworks it's in /etc/security/keytabs/* so it would be good if you clearly stated.
Please include the description of your cluster too like HA or not I see from the log failover to rm16 which suggest you have RM HA? 

Has the user kinited  before attempting the operation. Is user impersonating cabhbwg 

Happy hadooping