Explorer
Posts: 7
Registered: ‎07-27-2015
Spark jobs fail communicating with Hbase in a kerberos cluster
[ Edited ]

We have kerberos enabled in the our dev cluster.. Hadoop has no issues reading/writing to Hbase but Spark on YARN throws all kinds of expection when it tries to read/write from hbase.

 

15/08/11 13:07:47 WARN security.UserGroupInformation: PriviledgedActionException as:usera (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/11 13:07:47 WARN ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

15/08/11 13:07:47 ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)

at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:605)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:154)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:731)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:728)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

 

 

This is our Hbase code:

 

 

  // Instantiating Configuration class

        Configuration config = HBaseConfiguration.create();

        

        System.setProperty("java.security.krb5.conf", "/etc/krb5.conf");

        System.setProperty("java.security.auth.login.config", "/etc/zookeeper/conf/jaas.conf");

        

        config.set("hbase.zookeeper.quorum", "host-node2");

        config.set("hbase.zookeeper.property.clientPort", "2181");

 

        UserGroupInformation.setConfiguration(config);

        

        UserGroupInformation.loginUserFromKeytab("usera@DOMAIN.COM", "/home/ubuntu/usera.keytab");

 

 

        // Instantiating HTable class

        HTable hTable = new HTable(config, "test");

        Put p = new Put(Bytes.toBytes("row100"));

 

        // adding values using add() method

        // accepts column family name, qualifier/row name ,value

                

        p.add(Bytes.toBytes("cf1"),

        Bytes.toBytes(columnQualifier),Bytes.toBytes(s));

 

        System.out.println("Added row to Hbase table");

        // Saving the put Instance to the HTable.

        hTable.put(p);

        hTable.close();

 

 

I've given permissions for usera to read/write from test table

 

 

What seem to be causing this? Any help much appreciated.

Who Me Too'd this topic