Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

How to connect Storm to HBase ?

Labels:
avatar
author-rank Jagatheeshr author-rank
Guru

Created ‎10-08-2015 04:59 PM

Storm connectivity to HBase fails with "INFO: Could not update Hbase java.lang.NullPointerException"

Looks like Hbase,HDFS and Core site needs to be part of CLASSPATH.

Would like to know where exactly it should be.

Running storm classpath command shows below but doesnt show the *.*xml files.

storm classpath
/usr/hdp/2.2.6.0-2800/storm/lib/commons-configuration-1.10.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-common-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.2.6.0-2800/storm/lib/objenesis-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/slf4j-api-1.6.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/minlog-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-cred-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-client-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-server-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jersey-bundle-1.17.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-equality-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-exec-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlet-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/carbonite-1.4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-common-2.6.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/math.numeric-tower-0.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jgrapht-core-0.9.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-core-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-security-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-io-2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.2.6.0-2800/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/chill-java-0.3.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.macro-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.cli-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-fileupload-1.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/kryo-2.21.jar:/usr/hdp/2.2.6.0-2800/storm/lib/asm-4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-lang-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/storm-core-0.9.3.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/json-simple-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/snakeyaml-1.11.jar:/usr/hdp/2.2.6.0-2800/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clojure-1.5.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/guava-11.0.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-anti-forgery-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-random-1.2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlets-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/joda-time-2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-audit-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-io-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-impl-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-util-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clout-1.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-time-0.4.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-http-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/zookeeper.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-continuation-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-stacktrace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gson-2.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-storm-plugin-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-logging-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-classic-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-auth-2.4.0.jar

Browsing the internet shows that the files should be under .storm. Hence created the .storm directory and copied the *.*xml files to .storm but still no luck .

Any ideas ?

3,255 Views
1 ACCEPTED SOLUTION

avatar
author-rank abajwa author-rank
Guru

Created ‎10-08-2015 05:21 PM

There are steps and code for a working Kafka to Storm to HBase example on HDP 2.3 at the 3-part tutorial series here which may help:

http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/

http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/

http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/

In the sample code provided above, the hbase-site.xml was packaged into the uber jar by adding the below in the pom.xml

     <resources>
      <resource>
        <directory>/etc/hbase/conf</directory>
        <includes>
          <include>hbase-site.xml</include>
        </includes> 
      </resource>
      <resource>
        <directory>/opt/TruckEvents/Tutorials-master/src/main/resources</directory>
      </resource>      
    </resources>

View solution in original post

2,198 Views
4 REPLIES 4

avatar
author-rank abajwa author-rank
Guru

Created ‎10-08-2015 05:21 PM

There are steps and code for a working Kafka to Storm to HBase example on HDP 2.3 at the 3-part tutorial series here which may help:

http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/

http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/

http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/

In the sample code provided above, the hbase-site.xml was packaged into the uber jar by adding the below in the pom.xml

     <resources>
      <resource>
        <directory>/etc/hbase/conf</directory>
        <includes>
          <include>hbase-site.xml</include>
        </includes> 
      </resource>
      <resource>
        <directory>/opt/TruckEvents/Tutorials-master/src/main/resources</directory>
      </resource>      
    </resources>
2,199 Views

avatar
author-rank rohan_kapadekar
Explorer

Created ‎02-09-2016 07:28 AM

@Ali Bajwa unfortunately,none of the links works..

could you provide me with working links or a solution?

I too need to integrate Kafka data into HBase using Storm.

2,198 Views

avatar
author-rank phargis
Guru

Created ‎02-17-2016 04:19 PM

From the Storm TruckEvents tutorial, here is the Hbase pom reference for inclusion of jar files into the Storm deploy-jar. Beware that this is for HDP 2.2 example:

...
	<hbase.version>0.98.0.2.1.1.0-385-hadoop2</hbase.version>
...

	<!-- HBase Dependcies -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>log4j-over-slf4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Therefore, you need to include both the Hbase jars in the uber Storm jar, as well as the config file as shown in Ali's answer above.

2,198 Views

avatar
author-rank vvaks
Guru

Created ‎03-28-2016 06:06 AM

It's all about maven dependencies. 

<dependencies>

    <dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-core</artifactId>

		<version>0.10.0</version>

		<scope>provided</scope>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

    </dependency>

    <dependency>

		<groupId>org.apache.zookeeper</groupId>

		<artifactId>zookeeper</artifactId>

		<version>3.4.6</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hdfs</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-kafka</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

	<groupId>org.apache.kafka</groupId>

	<artifactId>kafka_2.10</artifactId>

	<version>0.8.2.2</version>

	<exclusions>

                <exclusion>

                    <groupId>org.apache.zookeeper</groupId>

                    <artifactId>zookeeper</artifactId>

                </exclusion>

                <exclusion>

                    <groupId>log4j</groupId>

                    <artifactId>log4j</artifactId>

                </exclusion>

            </exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hbase</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

    <dependency>

		<groupId>org.apache.hbase</groupId>

		<artifactId>hbase-client</artifactId>

		<version>1.1.1</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>	

	</dependency>
</dependencies>
2,198 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
View More Announcements