2019-09-28 16:55:30,706 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2019-09-28 16:55:30,917 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1568788887141_0253, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@201a4587) 2019-09-28 16:55:30,918 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: RM_DELEGATION_TOKEN, Service: 10.10.1.25:8032, Ident: (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) 2019-09-28 16:55:30,967 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2019-09-28 16:55:31,522 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data/yarn/nm/usercache/hdfs/appcache/application_1568788887141_0253   2019-09-30 15:19:00,539 [Yarn application state monitor] WARN org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache 2019-09-30 15:19:00,603 [Yarn application state monitor] WARN org.apache.hadoop.ipc.Client - Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache 2019-09-30 15:19:00,603 [Yarn application state monitor] WARN org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache 2019-09-30 15:19:00,683 [Yarn application state monitor] ERROR org.apache.spark.deploy.yarn.Client - Failed to contact YARN for application application_1568788887141_0254. org.apache.hadoop.security.token.SecretManager$InvalidToken: token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:190) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy18.getApplicationReport(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:408) at org.apache.spark.deploy.yarn.Client.getApplicationReport(Client.scala:316) at org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:1066) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:105) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1569660905008, maxDate=1570265705008, sequenceNumber=7653, masterKeyId=12) can't be found in cache at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1409) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy17.getApplicationReport(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187) ... 10 more 2019-09-30 15:19:00,690 [Yarn application state monitor] ERROR org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Yarn application has already exited with state FAILED! ... View more