Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

SOLVED Go to solution
Highlighted

Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

I've noticed that 24 hours after starting, the Spark Thrift server encounters authentication failures.

By the way i am on HDP 2.4.2

[Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 
The message directly before this (usually several hours beforehand) is
WARN UserGroupInformation: Exception encountered while running the renewal command. Aborting renew thread. ExitCodeException exitCode=1: kinit: Ticket expired while renewing credentials
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

Rising Star

Hi @Sandeep Nemuri, Can you please verify if spark.yarn.keytab and spark.yarn.principal is set in spark-thrift-sparkconf. This is a know bug. As STS is failing to renew the token once the cache expires, and there is no renew thread. Please add above properties and let me know if you are still seeing any issue.Thx

3 REPLIES 3

Re: Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

Rising Star

Hi @Sandeep Nemuri, Can you please verify if spark.yarn.keytab and spark.yarn.principal is set in spark-thrift-sparkconf. This is a know bug. As STS is failing to renew the token once the cache expires, and there is no renew thread. Please add above properties and let me know if you are still seeing any issue.Thx

Re: Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

thanks @nyadav i don't see the above two properties in spark-thrift-sparkconf. Adding them helped in resolving the issue.

Re: Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

Explorer

Is the issue resolved in HDP 2.6.4 ? We are facing similar issue on Spark2 History server.

Kerberos ticket renewal is failing and History server Web UI is inaccessible to due expired kerberos tgt.