Expert Contributor
Posts: 185
Registered: ‎07-01-2015

Spark with Hbase - Kerberos TGT

I am using Spark 1.6  (CDH 5.7) to connect to Hbase in a secured kerberized cluster. I observed a wierd behaviour, my jobs are sometimes failing with error message:


Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

The usual root cause for this kind of error is that the client does not have a valid kerberos ticket. But in my case the jobs fails almost randomly. What I observed that if I use dynamic allocation and the initial list of executors are lost (for example driver waiting for something) and then a new set of executors kicks in, and the program continues and tries to fetch something from Hbase, in the Executor logs I see these errors. 


If I turn off the dynamic allocation - so every task is executed by the initial list of provided executors, then it seems to me ok.  Never had a time to fully test it out (because the job is long running), but I assume that there is a correlation between the "pause" of the old and new executors, because I had sucesfull runs, where the new executors were on the fly allocated and they did not have a problem to access the hbase.


Any help appreciated,





Posts: 6
Registered: ‎08-06-2017

Re: Spark with Hbase - Kerberos TGT

if you are able to access HBase from Spark code. Could you help me? Please check the below url