Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Yarn issue

avatar
New Contributor

Hello Cloudera community, 

 

Could you please help me about Yarn issue below ? I'm new in Hadoop and I don't know very well hadoop cluster.

21/01/04 11:21:41 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm16
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528130, maxDate=1608841328130, sequenceNumber=288, masterKeyId=735) can't be found in cache
21/01/04 11:21:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528130, maxDate=1608841328130, sequenceNumber=288, masterKeyId=735) can't be found in cache
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528130, maxDate=1608841328130, sequenceNumber=288, masterKeyId=735) can't be found in cache
Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.
21/01/04 11:21:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/01/04 11:21:41 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
21/01/04 11:21:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
21/01/04 11:21:41 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
21/01/04 11:21:41 WARN security.UserGroupInformation: PriviledgedActionException as:user/server (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (token for cabhbwg: HDFS_DELEGATION_TOKEN owner=cabhbwg, renewer=yarn, realUser=user/server, issueDate=1608236528002, maxDate=1608841328002, sequenceNumber=32002, masterKeyId=1523) can't be found in cache
	at org.apache.hadoop.ipc.Client.call(Client.java:1504)
	at org.apache.hadoop.ipc.Client.call(Client.java:1441)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy15.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:588)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy16.getListing(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2145)
	at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:211)
	at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:198)
	at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:180)
	at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:180)
	at org.apache.hadoop.fs.FileContext$21.next(FileContext.java:1468)
	at org.apache.hadoop.fs.FileContext$21.next(FileContext.java:1463)
	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
	at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1463)
	at org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAllContainersLogs(LogCLIHelpers.java:156)
	at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:133)
	at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:186)

 

Thank you very much in advance. 

1 REPLY 1

avatar
Master Mentor

@ibrahima 

This community helps in 2 of the most used Hadoop flavors Cloudera and Hortonworks and these 2 software vendors handled and configured differently their Kerberos.

 

In cloudera the keytabs are found in /run/cloudera-scm-agent/process/* while in hortonworks it's in /etc/security/keytabs/* so it would be good if you clearly stated.
Please include the description of your cluster too like HA or not I see from the log failover to rm16 which suggest you have RM HA? 

Has the user kinited  before attempting the operation. Is user impersonating cabhbwg 

Happy hadooping