Jobs are getting failed to renew token TIMELINE_DELEGATION_TOKEN

[#1]

Hi,

We have a Kerberized Hadoop cluster. This is been running a for a while and we don't have any issues.

Suddenly all our weekly batch jobs failed with Token delegation key error.

Here is the error message

Connected to: Apache Hive (version 1.2.1000.2.4.2.0-258)
Driver: Hive JDBC (version 1.2.1000.2.4.2.0-258)
Transaction isolation: TRANSACTION_REPEATABLE_READ
INFO  : Tez session hasn't been created yet. Opening session
ERROR : Failed to execute tez graph.
org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1485103343268_0137 to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.19.8.245:8190, Ident: (owner=hive, renewer=yarn, realUser=, issueDate=1486486486195, maxDate=1487091286195, sequenceNumber=5130, masterKeyId=627)
        at org.apache.tez.client.TezClient.start(TezClient.java:414)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:271)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:151)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:75)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1485103343268_0137 to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.19.8.245:8190, Ident: (owner=hive, renewer=yarn, realUser=, issueDate=1486486486195, maxDate=1487091286195, sequenceNumber=5130, masterKeyId=627)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271)
        at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
        at org.apache.tez.client.TezClient.start(TezClient.java:409)
        ... 6 more
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)

It would be great if anyone has any idea what is triggering this issue all of a sudden

Thanks

Arun

[#2 · Accepted solution]

For the benefits for others here , after working with @Aruna dadi this was due to change in /etc/resolv.conf. Reverted the changes to file on all the nodes and restarted the HDFS, YARN and Hive services from Ambari.

[#3]

do kinit list with the keytab file and resolve the hostname mentioned in the keytab file. ensure it get resolved either by dns or through /etc/hosts (entry)