Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

avatar

I've noticed that 24 hours after starting, the Spark Thrift server encounters authentication failures.

By the way i am on HDP 2.4.2

[Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 
The message directly before this (usually several hours beforehand) is
WARN UserGroupInformation: Exception encountered while running the renewal command. Aborting renew thread. ExitCodeException exitCode=1: kinit: Ticket expired while renewing credentials
1 ACCEPTED SOLUTION

avatar
Expert Contributor

Hi @Sandeep Nemuri, Can you please verify if spark.yarn.keytab and spark.yarn.principal is set in spark-thrift-sparkconf. This is a know bug. As STS is failing to renew the token once the cache expires, and there is no renew thread. Please add above properties and let me know if you are still seeing any issue.Thx

View solution in original post

4 REPLIES 4

avatar
Expert Contributor

Hi @Sandeep Nemuri, Can you please verify if spark.yarn.keytab and spark.yarn.principal is set in spark-thrift-sparkconf. This is a know bug. As STS is failing to renew the token once the cache expires, and there is no renew thread. Please add above properties and let me know if you are still seeing any issue.Thx

avatar

thanks @nyadav i don't see the above two properties in spark-thrift-sparkconf. Adding them helped in resolving the issue.

avatar
Explorer

What's the name of the bug/ticket? Is it solved?

avatar

Is the issue resolved in HDP 2.6.4 ? We are facing similar issue on Spark2 History server.

Kerberos ticket renewal is failing and History server Web UI is inaccessible to due expired kerberos tgt.