Cloudera Labs
Provide feedback on Cloudera Labs
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

New in Cloudera Labs: SparkOnHBase

Master Collaborator

Announcing the newest component in Cloudera Labs: SparkOnHBase


SparkOnHBase is a simple reusable library for working with HBase and Spark. Among other things, it makes HBase connections seamless, allows you to do any combination of HBase operations on an RDD, and more.


Read more here.


New Contributor

Have you guys been able to figure out how to get this working on a secure HBase in --deploy-mode cluster ?

I've only been able to get it to work in client mode.


In cluster mode since the driver runs in the AM, I don't see any way for it to acquire the right credentials.



Cloudera Employee

Hey @ramblingpolak,


I would have to see the exception.  The only issue I can think of off hand would be a configuration difference from the edge node to the datanode.


Send the output of yarn logs -applicationId XYZ and I will see what I can do.


Ted Malaska

New Contributor

I've found a way to make this work. But first, the problem that I ran into...


When running spark with --deploy-mode client everything is kosher because you have run kinit on your edge node and have a valid TGT. The driver code (your Spark job) runs on the edge node and is able to obtain a delegation token for HBase. These credentials are then sent over to executors in the broadcast variable. 


However, when you run spark with --deploy-mode cluster your driver code is run within the application master which is launched on some nodemanager. This means that  you'll get the error below when you hit the code trying to request an HBase delegation token because you're not authenticated on the nodemanager your driver code happens to launch on.


This is avoided when accessing things like HDFS because the org.apache.spark.deploy.yarn.Client (which submits your job to YARN) is aware of HDFS and requests the proper tokens. It does not do this for HBase. By the time your driver code is reached, it's too late because its no longer running on the node you're running spark-submit on.


15/01/15 22:45:49 WARN security.UserGroupInformation: PriviledgedActionException as:adam (auth:SIMPLE) GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/01/15 22:45:49 WARN ipc.RpcClient: Exception encountered while connecting to the server : GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
15/01/15 22:45:49 ERROR ipc.RpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(
	at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(
	at org.apache.hadoop.hbase.ipc.RpcClient$Connection$
	at org.apache.hadoop.hbase.ipc.RpcClient$Connection$
	at Method)
	at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(
	at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(
	at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(
	at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(
	at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(
	at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(
	at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(
	at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(
	at Method)


So, without modifying the spark YARN client and making it HBase aware the only solution I found was to rely on for each user who wishes to submit a Spark YARN HBase app they must distribute a keytab to each node running a nodemanager (e.g. via ansible / scp).


In the you can then authenticate via the keytab to get a TGT and thus are allowed to obtain the proper HBase tokens.

Cloudera Employee

Cool, so the problem is I'm getting the key tab in the driver and the key tab is not on the datanode.  I will look into how to get the token given to Spark through spark-submit.



Cloudera Employee

So I updated the code today to work with 5.3 and made a work around for broadcast variables breaking in Spark Streaming.  But I haven't been able to figure out how to get the HBase creds without running in client mode.


I'm still working on it.  Let me know if you have any ideas.

New Contributor

My solution was to ensure that the user you're running as has an accessible kerberos keytab on each nodemanager.

e.g. /home/adam/adam.keytab on all machines running a nodemanager.
In that case i've modified my code to rely on the api for keytab auth to provide credentials..
Snippet below. This is part of the Kiji Spark integration I'm working on and should be open sourced in the next couple weeks
/** Provides Kiji-specific methods on `SparkContext` */
class SparkContextFunctions(@transient val sc: SparkContext) extends Serializable {

  import SparkContextFunctions._

  /** Returns a view of a Kiji table as `KijiRDD[T]`.
    * This method is made available on `SparkContext` by importing `org.kiji.spark._`
    * @param uri A KijiURI.
    * @param dataRequest A KijiDataRequest.
    * @param vClass ??? Need to talk to Adam.
    * @return An instance of a KijiRDD.
  def kijiRDD[T](uri: KijiURI, dataRequest: KijiDataRequest, vClass: Class[_ <: T]): KijiRDD[T] = {
    val authMode = sc.hadoopConfiguration.get("")"Running with $authMode authentication.")


    val sparkConf = sc.getConf

    val kerberosUsername = sparkConf.getOption("spark.kiji.kerberos.username")
    val keytab = sparkConf.getOption("spark.kiji.kerberos.keytab")

    // If the user specified both properties, then attempt to authenticate
    val ugi = if (kerberosUsername.nonEmpty && keytab.nonEmpty) {
      val ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(
      // Even if we authenticated, only request a token if security is enabled.
      if (UserGroupInformation.isSecurityEnabled) {
        TokenUtil.obtainAndCacheToken(sc.hadoopConfiguration, ugi)"Obtained and cached auth token for HBase.")
    } else {
      // Otherwise assume we are either on a non-secure cluster or the HBase auth token
      // has already been cached by the user.

    val credentials = ugi.getCredentials
    KijiRDD(sc, sc.hadoopConfiguration, credentials, uri, dataRequest).asInstanceOf[KijiRDD[T]]