08-03-2017 06:05 AM
I am using Spark 1.6 (CDH 5.7) to connect to Hbase in a secured kerberized cluster. I observed a wierd behaviour, my jobs are sometimes failing with error message:
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
The usual root cause for this kind of error is that the client does not have a valid kerberos ticket. But in my case the jobs fails almost randomly. What I observed that if I use dynamic allocation and the initial list of executors are lost (for example driver waiting for something) and then a new set of executors kicks in, and the program continues and tries to fetch something from Hbase, in the Executor logs I see these errors.
If I turn off the dynamic allocation - so every task is executed by the initial list of provided executors, then it seems to me ok. Never had a time to fully test it out (because the job is long running), but I assume that there is a correlation between the "pause" of the old and new executors, because I had sucesfull runs, where the new executors were on the fly allocated and they did not have a problem to access the hbase.
Any help appreciated,
08-06-2017 11:26 AM
if you are able to access HBase from Spark code. Could you help me? Please check the below url