Reply
Expert Contributor
Posts: 152
Registered: ‎07-01-2015

Spark with Hbase - Kerberos TGT

I am using Spark 1.6  (CDH 5.7) to connect to Hbase in a secured kerberized cluster. I observed a wierd behaviour, my jobs are sometimes failing with error message:

 

Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

The usual root cause for this kind of error is that the client does not have a valid kerberos ticket. But in my case the jobs fails almost randomly. What I observed that if I use dynamic allocation and the initial list of executors are lost (for example driver waiting for something) and then a new set of executors kicks in, and the program continues and tries to fetch something from Hbase, in the Executor logs I see these errors. 

 

If I turn off the dynamic allocation - so every task is executed by the initial list of provided executors, then it seems to me ok.  Never had a time to fully test it out (because the job is long running), but I assume that there is a correlation between the "pause" of the old and new executors, because I had sucesfull runs, where the new executors were on the fly allocated and they did not have a problem to access the hbase.

 

Any help appreciated,

Thanks

 

 

 

Highlighted
Explorer
Posts: 6
Registered: ‎08-06-2017

Re: Spark with Hbase - Kerberos TGT

if you are able to access HBase from Spark code. Could you help me? Please check the below url

 

https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/HBase-access-in-Spark-code-in-a-Ke...

 

Announcements