Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

[#1]

I've noticed that 24 hours after starting, the Spark Thrift server encounters authentication failures.

By the way i am on HDP 2.4.2

[Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 

The message directly before this (usually several hours beforehand) is

WARN UserGroupInformation: Exception encountered while running the renewal command. Aborting renew thread. ExitCodeException exitCode=1: kinit: Ticket expired while renewing credentials

[#2 · Accepted solution]

Hi @Sandeep Nemuri, Can you please verify if spark.yarn.keytab and spark.yarn.principal is set in spark-thrift-sparkconf. This is a know bug. As STS is failing to renew the token once the cache expires, and there is no renew thread. Please add above properties and let me know if you are still seeing any issue.Thx

[#3]

thanks @nyadav i don't see the above two properties in spark-thrift-sparkconf. Adding them helped in resolving the issue.

[#4]

What's the name of the bug/ticket? Is it solved?

[#5]

Is the issue resolved in HDP 2.6.4 ? We are facing similar issue on Spark2 History server.

Kerberos ticket renewal is failing and History server Web UI is inaccessible to due expired kerberos tgt.