question Re: How to connect Storm to HBase ? in Archives of Support Questions (Read Only)

How to connect Storm to HBase ?

Jagatheeshr — Thu, 08 Oct 2015 23:59:10 GMT

Storm connectivity to HBase fails with "INFO: Could not update Hbase java.lang.NullPointerException"

Looks like Hbase,HDFS and Core site needs to be part of CLASSPATH.

Would like to know where exactly it should be.

Running storm classpath command shows below but doesnt show the *.*xml files.

storm classpath
/usr/hdp/2.2.6.0-2800/storm/lib/commons-configuration-1.10.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-common-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.2.6.0-2800/storm/lib/objenesis-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/slf4j-api-1.6.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/minlog-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-cred-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-client-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-server-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jersey-bundle-1.17.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-equality-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-exec-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlet-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/carbonite-1.4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-common-2.6.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/math.numeric-tower-0.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jgrapht-core-0.9.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-core-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-security-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-io-2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.2.6.0-2800/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/chill-java-0.3.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.macro-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.cli-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-fileupload-1.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/kryo-2.21.jar:/usr/hdp/2.2.6.0-2800/storm/lib/asm-4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-lang-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/storm-core-0.9.3.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/json-simple-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/snakeyaml-1.11.jar:/usr/hdp/2.2.6.0-2800/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clojure-1.5.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/guava-11.0.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-anti-forgery-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-random-1.2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlets-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/joda-time-2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-audit-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-io-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-impl-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-util-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clout-1.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-time-0.4.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-http-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/zookeeper.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-continuation-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-stacktrace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gson-2.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-storm-plugin-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-logging-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-classic-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-auth-2.4.0.jar

Browsing the internet shows that the files should be under .storm. Hence created the .storm directory and copied the *.*xml files to .storm but still no luck .

Any ideas ?

Re: How to connect Storm to HBase ?

abajwa — Fri, 09 Oct 2015 00:21:55 GMT

There are steps and code for a working Kafka to Storm to HBase example on HDP 2.3 at the 3-part tutorial series here which may help:

http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/

http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/

http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/

In the sample code provided above, the hbase-site.xml was packaged into the uber jar by adding the below in the pom.xml

     <resources>
      <resource>
        <directory>/etc/hbase/conf</directory>
        <includes>
          <include>hbase-site.xml</include>
        </includes> 
      </resource>
      <resource>
        <directory>/opt/TruckEvents/Tutorials-master/src/main/resources</directory>
      </resource>      
    </resources>

Re: How to connect Storm to HBase ?

rohan_kapadekar — Tue, 09 Feb 2016 15:28:39 GMT

@Ali Bajwa unfortunately,none of the links works..

could you provide me with working links or a solution?

I too need to integrate Kafka data into HBase using Storm.

Re: How to connect Storm to HBase ?

phargis — Thu, 18 Feb 2016 00:19:09 GMT

From the Storm TruckEvents tutorial, here is the Hbase pom reference for inclusion of jar files into the Storm deploy-jar. Beware that this is for HDP 2.2 example:

...
	<hbase.version>0.98.0.2.1.1.0-385-hadoop2</hbase.version>
...

	<!-- HBase Dependcies -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>log4j-over-slf4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Therefore, you need to include both the Hbase jars in the uber Storm jar, as well as the config file as shown in Ali's answer above.

Re: How to connect Storm to HBase ?

vvaks — Mon, 28 Mar 2016 13:06:49 GMT

It's all about maven dependencies.

<dependencies>

    <dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-core</artifactId>

		<version>0.10.0</version>

		<scope>provided</scope>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

    </dependency>

    <dependency>

		<groupId>org.apache.zookeeper</groupId>

		<artifactId>zookeeper</artifactId>

		<version>3.4.6</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hdfs</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-kafka</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

	<groupId>org.apache.kafka</groupId>

	<artifactId>kafka_2.10</artifactId>

	<version>0.8.2.2</version>

	<exclusions>

                <exclusion>

                    <groupId>org.apache.zookeeper</groupId>

                    <artifactId>zookeeper</artifactId>

                </exclusion>

                <exclusion>

                    <groupId>log4j</groupId>

                    <artifactId>log4j</artifactId>

                </exclusion>

            </exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hbase</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

    <dependency>

		<groupId>org.apache.hbase</groupId>

		<artifactId>hbase-client</artifactId>

		<version>1.1.1</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>	

	</dependency>
</dependencies>