Reply
Explorer
Posts: 7
Registered: ‎07-27-2015

Spark jobs fail communicating with Hbase in a kerberos cluster

[ Edited ]

We have kerberos enabled in the our dev cluster.. Hadoop has no issues reading/writing to Hbase but Spark on YARN throws all kinds of expection when it tries to read/write from hbase.

 

15/08/11 13:07:47 WARN security.UserGroupInformation: PriviledgedActionException as:usera (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/11 13:07:47 WARN ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/11 13:07:47 ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)

at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:605)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:154)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:731)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:728)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

 

 

This is our Hbase code:

 

 

  // Instantiating Configuration class

        Configuration config = HBaseConfiguration.create();

        

        System.setProperty("java.security.krb5.conf", "/etc/krb5.conf");

        System.setProperty("java.security.auth.login.config", "/etc/zookeeper/conf/jaas.conf");

        

        config.set("hbase.zookeeper.quorum", "host-node2");

        config.set("hbase.zookeeper.property.clientPort", "2181");

 

        UserGroupInformation.setConfiguration(config);

        

        UserGroupInformation.loginUserFromKeytab("usera@DOMAIN.COM", "/home/ubuntu/usera.keytab");

 

 

        // Instantiating HTable class

        HTable hTable = new HTable(config, "test");

        Put p = new Put(Bytes.toBytes("row100"));

 

        // adding values using add() method

        // accepts column family name, qualifier/row name ,value

                

        p.add(Bytes.toBytes("cf1"),

        Bytes.toBytes(columnQualifier),Bytes.toBytes(s));

 

        System.out.println("Added row to Hbase table");

        // Saving the put Instance to the HTable.

        hTable.put(p);

        hTable.close();

 

 

I've given permissions for usera to read/write from test table

 

 

What seem to be causing this? Any help much appreciated.

Posts: 1,567
Kudos: 289
Solutions: 240
Registered: ‎07-31-2013

Re: Spark jobs fail communicating with Hbase in a kerberos cluster

Given distributed-work, HBase clients will require to work with tokens instead of keytabs to be able to authenticate properly.

See http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ and the sources of the project to understand how this is done.
Backline Customer Operations Engineer
Contributor
Posts: 43
Registered: ‎07-27-2015

Re: Spark jobs fail communicating with Hbase in a kerberos cluster

[ Edited ]

HI Harsh,

This is very very urgent for me. I would appreciate it if you could give me a bit of good advice.

we are working with CDH 5.7.3 kerberos cluster. I faced the same problem that spark job fail write to hbase.

and I read the blog that you mentioned.

So, I change the code using hbase-spark.jar. I got the below error when runing spark with cluster mode.

 

16/12/14 10:31:05 ERROR ipc.RpcClientImpl: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

So, I would like to know is this an open issue ?:  https://github.com/cloudera-labs/SparkOnHBase/issues/6 

Or was I some way errors?

Or I how to accecss hbase with spark in secrity cluster?

 

Thanks

BR

Paul

 

Posts: 1,567
Kudos: 289
Solutions: 240
Registered: ‎07-31-2013

Re: Spark jobs fail communicating with Hbase in a kerberos cluster

That specific repository is defunct. Its work was contributed upstream and is available as part of HBase itself since CDH 5.7.0 and the upstream work is newer than what is in that repo.

I cannot point what is wrong in your code that's acting out as the failure (as its not available in your post), but here's a few code examples on how to use the HBase's native spark component: https://github.com/cloudera/hbase/tree/cdh5.7.1-release/hbase-spark/src/main/java/org/apache/hadoop/...

The code presented in the above will work in both secure and insecure modes. To give the examples a try to check this, you can invoke them directly. For ex:

~> kinit … # Or do a kdestroy to force an error…
~> hbase shell
> create 't', 'f'
> ^D
~> SPARK_CLASSPATH=$(hbase classpath) spark-submit --class org.apache.hadoop.hbase.spark.example.hbasecontext.JavaHBaseBulkPutExample t f
~> SPARK_CLASSPATH=$(hbase classpath) spark-submit --class org.apache.hadoop.hbase.spark.example.hbasecontext.JavaHBaseBulkGetExample t

Under the hood, the Spark connector in HBase (which works only on Spark 1.x BTW, not on Spark 2.x yet due to incompatible changes in the latter) obtains proper tokens from HBase in the driver/runner stage and then uses these tokens for authentication on the executor side, where a TGT is unavailable.

Modelling your implementation after the shown example sources should yield you the same behaviour. The implementation does not use a pure HBase Java API such as what the OP was using here originally.
Backline Customer Operations Engineer
Contributor
Posts: 43
Registered: ‎07-27-2015

Re: Spark jobs fail communicating with Hbase in a kerberos cluster

HI Harsh,

Thanks again.

The below is my code:

  def main(args: Array[String]) {

val tableName = "bd_recom:dw_users"
val columnFamily = "device"

val sparkConf = new SparkConf().setAppName("HBaseBulkPutExample " +
tableName + " " + columnFamily)
val sc = new SparkContext(sparkConf)

try {
//[(Array[Byte], Array[(Array[Byte], Array[Byte], Array[Byte])])]
val rdd = sc.parallelize(Array(
(Bytes.toBytes("11"),
Array((Bytes.toBytes(columnFamily), Bytes.toBytes("12"), Bytes.toBytes("1")))),
(Bytes.toBytes("22"),
Array((Bytes.toBytes(columnFamily), Bytes.toBytes("12"), Bytes.toBytes("2")))),
(Bytes.toBytes("33"),
Array((Bytes.toBytes(columnFamily), Bytes.toBytes("12"), Bytes.toBytes("3")))),
(Bytes.toBytes("44"),
Array((Bytes.toBytes(columnFamily), Bytes.toBytes("12"), Bytes.toBytes("4")))),
(Bytes.toBytes("55"),
Array((Bytes.toBytes(columnFamily), Bytes.toBytes("12"), Bytes.toBytes("5"))))
))

val conf = HBaseConfiguration.create()

val hbaseContext = new HBaseContext(sc, conf)
hbaseContext.bulkPut[(Array[Byte], Array[(Array[Byte], Array[Byte], Array[Byte])])](rdd,
TableName.valueOf(tableName),
(putRecord) => {
val put = new Put(putRecord._1)
putRecord._2.foreach((putValue) =>
put.addColumn(putValue._1, putValue._2, putValue._3))
put
});
} finally {
sc.stop()
}
}

there is my spark-submit command:

 

spark-submit --class com.userprofile.HiveToHbase2 --master yarn-cluster --queue root.plbd --driver-memory 2G --num-executors 4 --executor-memory 1G --executor-cores 2 --principal bd_recom@OD.BETA --keytab bd_recom.keytab sparksample-0.0.1-SNAPSHOT.jar

Unfortunately, It seem got the above issue.

 

Could you give me some advise?

BR

Paul

 

 

Posts: 1,567
Kudos: 289
Solutions: 240
Registered: ‎07-31-2013

Re: Spark jobs fail communicating with Hbase in a kerberos cluster

Do you perform a kinit prior to execution of the submit command? The keytab
argument you pass is for streaming jobs to receive new HDFS DTs
continually, but does not impact the driver.

Also, does the example run fine? Do you run it on a HBase Gateway host?
Backline Customer Operations Engineer
Contributor
Posts: 43
Registered: ‎07-27-2015

Re: Spark jobs fail communicating with Hbase in a kerberos cluster

[ Edited ]

HI Harsh:

 

At the first, I run the all spark jar on hbase gateway host, and got the same error.

The second , I cannot run the example that you mentioned.  

 

Thanks.

Paul

 

I run the command -----------------------
spark-submit --class org.apache.hadoop.hbase.spark.example.hbasecontext.JavaHBaseBulkPutExample t f

got the below error------------------------
Warning: Local jar /home/hadoop/t does not exist, skipping.
JavaHBaseBulkPutExample  {tableName} {columnFamily}

I run the command ----------------------------
 spark-submit --queue root.plbd --class org.apache.hadoop.hbase.spark.example.hbasecontext.JavaHBaseBulkPutExample --master yarn-cluster --driver-memory 2G --num-executors 4 --executor-memory 1G --executor-cores 2 hbase-spark-1.2.0-cdh5.7.3.jar bd_recom:dw_user device

I got the below info----------------------------
16/12/14 22:42:58 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on arch-od-data03.beta1.fn:43144 (size: 430.0 B, free: 530.2 MB)
16/12/14 22:42:58 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on arch-od-data05.beta1.fn:52828 (size: 430.0 B, free: 530.2 MB)
16/12/14 22:42:59 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2367 ms on arch-od-data04.beta1.fn (1/8)
16/12/14 22:42:59 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 2771 ms on arch-od-data03.beta1.fn (2/8)
16/12/14 22:42:59 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 2831 ms
on arch-od-data05.beta1.fn (3/8

since no zookeeper config, cannot access zookeeper
------------------------------------------
I put the --files /etc/hbase/conf/hbase-site.xml I got the same error that is below

16/12/14 22:55:45 INFO zookeeper.ClientCnxn: EventThread shut down
16/12/14 22:55:45 WARN security.UserGroupInformation: PriviledgedActionException as:bd_recom (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
16/12/14 22:55:45 WARN ipc.RpcClientImpl: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
16/12/14 22:55:45 ERROR ipc.RpcClientImpl: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:181)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)

So, I bold conjecture the yarn-cluster that isnot be supported with the cdh5.7.3 version

 

Contributor
Posts: 43
Registered: ‎07-27-2015

Re: Spark jobs fail communicating with Hbase in a kerberos cluster

HI Harsh:

Could you please help me to confirm :  that is not be supported with  kerberos  CDH5.7.3 cluster to use yarn-cluster command to submit the spark job that include the access hbase?

Thanks 

Paul

New Contributor
Posts: 2
Registered: ‎12-07-2015

Re: Spark jobs fail communicating with Hbase in a kerberos cluster

Hi,

 

I suppose to have exactly the same issue as described in the previous post, can you let me know if you have an answer.

 

Thanks

Announcements