Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

why does oozie launcher fail for the "RM_DELEGATION_TOKEN can't be found in cache" issue on a non-security cluster?

Highlighted

why does oozie launcher fail for the "RM_DELEGATION_TOKEN can't be found in cache" issue on a non-security cluster?

New Contributor

2019-09-28 16:55:30,706 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2019-09-28 16:55:30,917 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1568788887141_0253, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@201a4587)
2019-09-28 16:55:30,918 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: RM_DELEGATION_TOKEN, Service: 10.10.1.25:8032, Ident: (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12)
2019-09-28 16:55:30,967 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2019-09-28 16:55:31,522 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data/yarn/nm/usercache/hdfs/appcache/application_1568788887141_0253

 

2019-09-30 15:19:00,539 [Yarn application state monitor] WARN org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache
2019-09-30 15:19:00,603 [Yarn application state monitor] WARN org.apache.hadoop.ipc.Client - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache
2019-09-30 15:19:00,603 [Yarn application state monitor] WARN org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache
2019-09-30 15:19:00,683 [Yarn application state monitor] ERROR org.apache.spark.deploy.yarn.Client - Failed to contact YARN for application application_1568788887141_0254.
org.apache.hadoop.security.token.SecretManager$InvalidToken: token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:190)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy18.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:408)
at org.apache.spark.deploy.yarn.Client.getApplicationReport(Client.scala:316)
at org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:1066)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:105)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy17.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187)
... 10 more
2019-09-30 15:19:00,690 [Yarn application state monitor] ERROR org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Yarn application has already exited with state FAILED!

Don't have an account?
Coming from Hortonworks? Activate your account here