Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Solved Go to solution

How to connect Storm to HBase ?

Labels:
avatar
author-rank Jagatheeshr author-rank
Guru

Created ‎10-08-2015 04:59 PM

Storm connectivity to HBase fails with "INFO: Could not update Hbase java.lang.NullPointerException"

Looks like Hbase,HDFS and Core site needs to be part of CLASSPATH.

Would like to know where exactly it should be.

Running storm classpath command shows below but doesnt show the *.*xml files.

storm classpath
/usr/hdp/2.2.6.0-2800/storm/lib/commons-configuration-1.10.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-common-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.2.6.0-2800/storm/lib/objenesis-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/slf4j-api-1.6.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/minlog-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-cred-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-client-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-server-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jersey-bundle-1.17.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-equality-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-exec-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlet-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/carbonite-1.4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-common-2.6.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/math.numeric-tower-0.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jgrapht-core-0.9.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-core-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-security-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-io-2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.2.6.0-2800/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/chill-java-0.3.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.macro-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.cli-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-fileupload-1.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/kryo-2.21.jar:/usr/hdp/2.2.6.0-2800/storm/lib/asm-4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-lang-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/storm-core-0.9.3.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/json-simple-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/snakeyaml-1.11.jar:/usr/hdp/2.2.6.0-2800/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clojure-1.5.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/guava-11.0.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-anti-forgery-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-random-1.2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlets-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/joda-time-2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-audit-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-io-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-impl-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-util-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clout-1.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-time-0.4.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-http-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/zookeeper.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-continuation-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-stacktrace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gson-2.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-storm-plugin-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-logging-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-classic-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-auth-2.4.0.jar

Browsing the internet shows that the files should be under .storm. Hence created the .storm directory and copied the *.*xml files to .storm but still no luck .

Any ideas ?

2,669 Views
1 ACCEPTED SOLUTION

avatar
author-rank abajwa author-rank
Guru

Created ‎10-08-2015 05:21 PM

There are steps and code for a working Kafka to Storm to HBase example on HDP 2.3 at the 3-part tutorial series here which may help:

http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/

http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/

http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/

In the sample code provided above, the hbase-site.xml was packaged into the uber jar by adding the below in the pom.xml

     <resources>
      <resource>
        <directory>/etc/hbase/conf</directory>
        <includes>
          <include>hbase-site.xml</include>
        </includes> 
      </resource>
      <resource>
        <directory>/opt/TruckEvents/Tutorials-master/src/main/resources</directory>
      </resource>      
    </resources>

View solution in original post

1,612 Views
4 REPLIES 4

avatar
author-rank abajwa author-rank
Guru

Created ‎10-08-2015 05:21 PM

There are steps and code for a working Kafka to Storm to HBase example on HDP 2.3 at the 3-part tutorial series here which may help:

http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/

http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/

http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/

In the sample code provided above, the hbase-site.xml was packaged into the uber jar by adding the below in the pom.xml

     <resources>
      <resource>
        <directory>/etc/hbase/conf</directory>
        <includes>
          <include>hbase-site.xml</include>
        </includes> 
      </resource>
      <resource>
        <directory>/opt/TruckEvents/Tutorials-master/src/main/resources</directory>
      </resource>      
    </resources>
1,613 Views

avatar
author-rank rohan_kapadekar
Explorer

Created ‎02-09-2016 07:28 AM

@Ali Bajwa unfortunately,none of the links works..

could you provide me with working links or a solution?

I too need to integrate Kafka data into HBase using Storm.

1,612 Views

avatar
author-rank phargis
Guru

Created ‎02-17-2016 04:19 PM

From the Storm TruckEvents tutorial, here is the Hbase pom reference for inclusion of jar files into the Storm deploy-jar. Beware that this is for HDP 2.2 example:

...
	<hbase.version>0.98.0.2.1.1.0-385-hadoop2</hbase.version>
...

	<!-- HBase Dependcies -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>${hbase.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>log4j-over-slf4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Therefore, you need to include both the Hbase jars in the uber Storm jar, as well as the config file as shown in Ali's answer above.

1,612 Views

avatar
author-rank vvaks
Guru

Created ‎03-28-2016 06:06 AM

It's all about maven dependencies. 

<dependencies>

    <dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-core</artifactId>

		<version>0.10.0</version>

		<scope>provided</scope>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

    </dependency>

    <dependency>

		<groupId>org.apache.zookeeper</groupId>

		<artifactId>zookeeper</artifactId>

		<version>3.4.6</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hdfs</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-kafka</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

	<dependency>

	<groupId>org.apache.kafka</groupId>

	<artifactId>kafka_2.10</artifactId>

	<version>0.8.2.2</version>

	<exclusions>

                <exclusion>

                    <groupId>org.apache.zookeeper</groupId>

                    <artifactId>zookeeper</artifactId>

                </exclusion>

                <exclusion>

                    <groupId>log4j</groupId>

                    <artifactId>log4j</artifactId>

                </exclusion>

            </exclusions>

	</dependency>

	<dependency>

		<groupId>org.apache.storm</groupId>

		<artifactId>storm-hbase</artifactId>

		<version>0.10.0</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>

	</dependency>

    <dependency>

		<groupId>org.apache.hbase</groupId>

		<artifactId>hbase-client</artifactId>

		<version>1.1.1</version>

		<exclusions>

			<exclusion> 

    			<groupId>org.slf4j</groupId>

    			<artifactId>slf4j-log4j12</artifactId>

  			</exclusion>

  			<exclusion> 

    			<groupId>log4j</groupId>

    			<artifactId>log4j</artifactId>

  			</exclusion>

  		</exclusions>	

	</dependency>
</dependencies>
1,612 Views
0 Kudos
webinar banner
Announcements
Product Announcements
[ANNOUNCE] Cloudera on Public Cloud Runtime 7.2.18 is GA!
What's New @ Cloudera
Cloudera Operational Database (COD) supports enhancements to...
Community Announcements
March 2024 Community Highlights
Community Announcements
Celebrating 100,000 Members: A Journey of Growth and Connect...
Product Announcements
[ANNOUNCE] New Cloudera JDBC Connector for Apache Hive 2.6.2...
View More Announcements
meetups banner