question Re: Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts in Archives of Support Questions (Read Only)

Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

sandyy006 — Thu, 29 Dec 2016 19:54:53 GMT

I've noticed that 24 hours after starting, the Spark Thrift server encounters authentication failures.

By the way i am on HDP 2.4.2

[Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

The message directly before this (usually several hours beforehand) is

WARN UserGroupInformation: Exception encountered while running the renewal command. Aborting renew thread. ExitCodeException exitCode=1: kinit: Ticket expired while renewing credentials

Re: Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

nyadav — Thu, 29 Dec 2016 19:59:54 GMT

Hi @Sandeep Nemuri, Can you please verify if spark.yarn.keytab and spark.yarn.principal is set in spark-thrift-sparkconf. This is a know bug. As STS is failing to renew the token once the cache expires, and there is no renew thread. Please add above properties and let me know if you are still seeing any issue.Thx

Re: Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

sandyy006 — Fri, 30 Dec 2016 18:02:02 GMT

thanks @nyadav i don't see the above two properties in spark-thrift-sparkconf. Adding them helped in resolving the issue.

Re: Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

shankar_bagewad — Wed, 07 Aug 2019 20:13:59 GMT

Is the issue resolved in HDP 2.6.4 ? We are facing similar issue on Spark2 History server.

Kerberos ticket renewal is failing and History server Web UI is inaccessible to due expired kerberos tgt.

Re: Spark Thrift Servers are failing to renew TGTs automatically and require manual restarts

tribbloid — Thu, 20 Jan 2022 00:23:35 GMT

What's the name of the bug/ticket? Is it solved?