- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to connect Storm to HBase ?
- Labels:
-
Apache HBase
-
Apache Storm
Created ‎10-08-2015 04:59 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Storm connectivity to HBase fails with "INFO: Could not update Hbase java.lang.NullPointerException"
Looks like Hbase,HDFS and Core site needs to be part of CLASSPATH.
Would like to know where exactly it should be.
Running storm classpath command shows below but doesnt show the *.*xml files.
storm classpath /usr/hdp/2.2.6.0-2800/storm/lib/commons-configuration-1.10.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-common-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/2.2.6.0-2800/storm/lib/objenesis-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/slf4j-api-1.6.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/minlog-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-cred-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-client-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-server-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jersey-bundle-1.17.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-equality-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-exec-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlet-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/carbonite-1.4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-common-2.6.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/math.numeric-tower-0.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jgrapht-core-0.9.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-core-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-security-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-io-2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.2.6.0-2800/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/chill-java-0.3.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.macro-0.1.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/tools.cli-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-fileupload-1.2.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.2.6.0-2800/storm/lib/kryo-2.21.jar:/usr/hdp/2.2.6.0-2800/storm/lib/asm-4.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-lang-2.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/storm-core-0.9.3.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/json-simple-1.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/snakeyaml-1.11.jar:/usr/hdp/2.2.6.0-2800/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clojure-1.5.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/guava-11.0.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-anti-forgery-1.0.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/crypto-random-1.2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-servlets-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/joda-time-2.0.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-audit-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-io-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-plugins-impl-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-util-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clout-1.0.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-time-0.4.1.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-http-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/zookeeper.jar:/usr/hdp/2.2.6.0-2800/storm/lib/jetty-continuation-7.6.13.v20130916.jar:/usr/hdp/2.2.6.0-2800/storm/lib/clj-stacktrace-0.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/gson-2.2.4.jar:/usr/hdp/2.2.6.0-2800/storm/lib/ranger-storm-plugin-0.4.0.2.2.6.0-2800.jar:/usr/hdp/2.2.6.0-2800/storm/lib/commons-logging-1.2.jar:/usr/hdp/2.2.6.0-2800/storm/lib/logback-classic-1.0.6.jar:/usr/hdp/2.2.6.0-2800/storm/lib/hadoop-auth-2.4.0.jar
Browsing the internet shows that the files should be under .storm. Hence created the .storm directory and copied the *.*xml files to .storm but still no luck .
Any ideas ?
Created ‎10-08-2015 05:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There are steps and code for a working Kafka to Storm to HBase example on HDP 2.3 at the 3-part tutorial series here which may help:
http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/
http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/
http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/
In the sample code provided above, the hbase-site.xml was packaged into the uber jar by adding the below in the pom.xml
<resources> <resource> <directory>/etc/hbase/conf</directory> <includes> <include>hbase-site.xml</include> </includes> </resource> <resource> <directory>/opt/TruckEvents/Tutorials-master/src/main/resources</directory> </resource> </resources>
Created ‎10-08-2015 05:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There are steps and code for a working Kafka to Storm to HBase example on HDP 2.3 at the 3-part tutorial series here which may help:
http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/
http://hortonworks.com/hadoop-tutorial/ingesting-processing-real-time-events-apache-storm/
http://hortonworks.com/hadoop-tutorial/real-time-data-ingestion-hbase-hive-using-storm-bolt/
In the sample code provided above, the hbase-site.xml was packaged into the uber jar by adding the below in the pom.xml
<resources> <resource> <directory>/etc/hbase/conf</directory> <includes> <include>hbase-site.xml</include> </includes> </resource> <resource> <directory>/opt/TruckEvents/Tutorials-master/src/main/resources</directory> </resource> </resources>
Created ‎02-09-2016 07:28 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Ali Bajwa unfortunately,none of the links works..
could you provide me with working links or a solution?
I too need to integrate Kafka data into HBase using Storm.
Created ‎02-17-2016 04:19 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
From the Storm TruckEvents tutorial, here is the Hbase pom reference for inclusion of jar files into the Storm deploy-jar. Beware that this is for HDP 2.2 example:
... <hbase.version>0.98.0.2.1.1.0-385-hadoop2</hbase.version> ... <!-- HBase Dependcies --> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> <version>${hbase.version}</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>log4j-over-slf4j</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency>
Therefore, you need to include both the Hbase jars in the uber Storm jar, as well as the config file as shown in Ali's answer above.
Created ‎03-28-2016 06:06 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It's all about maven dependencies.
<dependencies> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-core</artifactId> <version>0.10.0</version> <scope>provided</scope> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.zookeeper</groupId> <artifactId>zookeeper</artifactId> <version>3.4.6</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-hdfs</artifactId> <version>0.10.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-kafka</artifactId> <version>0.10.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10</artifactId> <version>0.8.2.2</version> <exclusions> <exclusion> <groupId>org.apache.zookeeper</groupId> <artifactId>zookeeper</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-hbase</artifactId> <version>0.10.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> <version>1.1.1</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> </exclusions> </dependency> </dependencies>
