Support Questions
Find answers, ask questions, and share your expertise

Unable to push data from Storm to HBase in Kerberos enabled clusters

Unable to push data from Storm to HBase in Kerberos enabled clusters

I have written Spout which is pushing dummy data and bolt which is opening connection with HBase and updating data to HBase using phoenix driver as mentioned in below snippet.

    public void execute(Tuple tuple) {
        String name = tuple.getString(0);
        try {
            stmt.executeUpdate("upsert into StormWriter values ("+ id.toString() +",'" + tuple.getString(0) + " ')");
            con.commit();
            id++;
            outputCollector.ack(tuple);
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }


    public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
        this.outputCollector = outputCollector;
        try {
            con = DriverManager.getConnection("jdbc:phoenix:<ZK_QUORUM>:2181:/hbase-secure:<PRINCIPAL>:<KEYTAB>");
            stmt = con.createStatement();
            System.out.println("Connection Prepared successfully");
        } catch (SQLException e) {
            System.out.println("Failed to prepare Connection");
            e.printStackTrace();
        }
    }

Both HBase and Storm clusters are Kerberized.

HBase cluster version HDP 2.5.0

Storm Cluster version HDP 2.4.3

I am successfully able to submit the topology and spout is emitting the data but bolt is not able to push data as connection is failing with below error.

2018-01-17 09:26:18.322 o.a.h.h.i.AbstractRpcClient [Thread-5-Writer-bolt] [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2018-01-17 09:26:18.323 o.a.h.h.i.AbstractRpcClient [Thread-5-Writer-bolt] [ERROR] SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_40]
        at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) ~[stormjar.jar:?]
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:642) ~[stormjar.jar:?]
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:166) [stormjar.jar:?]
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:769) ~[stormjar.jar:?]
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:766) ~[stormjar.jar:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_40]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_40]