Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

How to connect Storm to HBase ?

Labels:
avatar
author-rank Jagatheeshr author-rank
Guru

Created ‎10-08-2015 04:59 PM

Storm connectivity to HBase fails with "INFO: Could not update Hbase java.lang.NullPointerException"

Looks like Hbase,HDFS and Core site needs to be part of CLASSPATH.

Would like to know where exactly it should be.

Running storm classpath command shows below but doesnt show the *.*xml files.

storm classpath
/usr/hdp/2.2.6.0-2800/storm/lib/commons-configuration-1.10.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-common-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.2.6.0-2800/storm/lib/objenesis-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/slf4j-api-1.6.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/minlog-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-cred-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-client-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-server-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jersey-bundle-1.17.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-equality-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-exec-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlet-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/carbonite-1.4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-common-2.6.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/math.numeric-tower-0.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jgrapht-core-0.9.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-core-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-security-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-io-2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.2.6.0-2800/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/chill-java-0.3.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.macro-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.cli-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-fileupload-1.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/kryo-2.21.jar:/usr/hdp/2.2.6.0-2800/storm/lib/asm-4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-lang-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/storm-core-0.9.3.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/json-simple-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/snakeyaml-1.11.jar:/usr/hdp/2.2.6.0-2800/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clojure-1.5.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/guava-11.0.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-anti-forgery-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-random-1.2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlets-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/joda-time-2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-audit-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-io-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-impl-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-util-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clout-1.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-time-0.4.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-http-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/zookeeper.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-continuation-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-stacktrace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gson-2.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-storm-plugin-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-logging-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-classic-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-auth-2.4.0.jar

Browsing the internet shows that the files should be under .storm. Hence created the .storm directory and copied the *.*xml files to .storm but still no luck .

Any ideas ?

3,170 Views
1 ACCEPTED SOLUTION

avatar
author-rank abajwa author-rank
Guru

Created ‎10-08-2015 05:21 PM

There are steps and code for a working Kafka to Storm to HBase example on HDP 2.3 at the 3-part tutorial series here which may help:

http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/

http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/

http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/

In the sample code provided above, the hbase-site.xml was packaged into the uber jar by adding the below in the pom.xml

     <resources>
      <resource>
        <directory>/etc/hbase/conf</directory>
        <includes>
          <include>hbase-site.xml</include>
        </includes> 
      </resource>
      <resource>
        <directory>/opt/TruckEvents/Tutorials-master/src/main/resources</directory>
      </resource>      
    </resources>

View solution in original post

2,113 Views
4 REPLIES 4

avatar
author-rank abajwa author-rank
Guru

Created ‎10-08-2015 05:21 PM

There are steps and code for a working Kafka to Storm to HBase example on HDP 2.3 at the 3-part tutorial series here which may help:

http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/

http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/

http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/

In the sample code provided above, the hbase-site.xml was packaged into the uber jar by adding the below in the pom.xml

     <resources>
      <resource>
        <directory>/etc/hbase/conf</directory>
        <includes>
          <include>hbase-site.xml</include>
        </includes> 
      </resource>
      <resource>
        <directory>/opt/TruckEvents/Tutorials-master/src/main/resources</directory>
      </resource>      
    </resources>
2,114 Views

avatar
author-rank rohan_kapadekar
Explorer

Created ‎02-09-2016 07:28 AM

@Ali Bajwa unfortunately,none of the links works..

could you provide me with working links or a solution?

I too need to integrate Kafka data into HBase using Storm.

2,113 Views

avatar
author-rank phargis
Guru

Created ‎02-17-2016 04:19 PM

From the Storm TruckEvents tutorial, here is the Hbase pom reference for inclusion of jar files into the Storm deploy-jar. Beware that this is for HDP 2.2 example:

...
	<hbase.version>0.98.0.2.1.1.0-385-hadoop2</hbase.version>
...

	<!-- HBase Dependcies -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>log4j-over-slf4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Therefore, you need to include both the Hbase jars in the uber Storm jar, as well as the config file as shown in Ali's answer above.

2,113 Views

avatar
author-rank vvaks
Guru

Created ‎03-28-2016 06:06 AM

It's all about maven dependencies. 

<dependencies>

    <dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-core</artifactId>

		<version>0.10.0</version>

		<scope>provided</scope>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

    </dependency>

    <dependency>

		<groupId>org.apache.zookeeper</groupId>

		<artifactId>zookeeper</artifactId>

		<version>3.4.6</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hdfs</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-kafka</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

	<groupId>org.apache.kafka</groupId>

	<artifactId>kafka_2.10</artifactId>

	<version>0.8.2.2</version>

	<exclusions>

                <exclusion>

                    <groupId>org.apache.zookeeper</groupId>

                    <artifactId>zookeeper</artifactId>

                </exclusion>

                <exclusion>

                    <groupId>log4j</groupId>

                    <artifactId>log4j</artifactId>

                </exclusion>

            </exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hbase</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

    <dependency>

		<groupId>org.apache.hbase</groupId>

		<artifactId>hbase-client</artifactId>

		<version>1.1.1</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>	

	</dependency>
</dependencies>
2,113 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera DataFlow 2.9 now supports building GenAI pipelines ...
What's New @ Cloudera
Accelerate Data Scientist Productivity with Cloudera Copilot...
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics - Kubernetes Operato...
Community Announcements
October 2024 Community Highlights
What's New @ Cloudera
Accelerate Your AI with the Cloudera AI Inference Service wi...
View More Announcements