Hi all,
we are developing a storm topology to write streaming data in hive database but the following errors occurs during executions:
1) Using Hive library version 1.2.1 (http://search.maven.org/#artifactdetails|org.apache.hive|hive|1.2.1|pom) and configuration as in the attached pom1.xml file, the error is:
43088 [Thread-12-hiveBolt] ERROR b.s.d.executor -
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoSuchFieldError: METASTORE_FILTER_HOOK
at org.apache.storm.hive.common.HiveWriter.callWithTimeout(HiveWriter.java:357) ~[StormTopology-0.1.jar:?]
at org.apache.storm.hive.common.HiveWriter.newConnection(HiveWriter.java:226) ~[StormTopology-0.1.jar:?]
at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:69) ~[StormTopology-0.1.jar:?]
at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[StormTopology-0.1.jar:?]
at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:219) ~[StormTopology-0.1.jar:?]
at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:102) [StormTopology-0.1.jar:?]
at backtype.storm.daemon.executor$fn__5694$tuple_action_fn__5696.invoke(executor.clj:690) [StormTopology-0.1.jar:?]
at backtype.storm.daemon.executor$mk_task_receiver$fn__5615.invoke(executor.clj:436) [StormTopology-0.1.jar:?]
at backtype.storm.disruptor$clojure_handler$reify__5189.onEvent(disruptor.clj:58) [StormTopology-0.1.jar:?]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:132) [StormTopology-0.1.jar:?]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:106) [StormTopology-0.1.jar:?]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [StormTopology-0.1.jar:?]
at backtype.storm.daemon.executor$fn__5694$fn__5707$fn__5758.invoke(executor.clj:819) [StormTopology-0.1.jar:?]
at backtype.storm.util$async_loop$fn__545.invoke(util.clj:479) [StormTopology-0.1.jar:?]
at clojure.lang.AFn.run(AFn.java:22) [StormTopology-0.1.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchFieldError: METASTORE_FILTER_HOOK
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_71]
at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_71]
at org.apache.storm.hive.common.HiveWriter.callWithTimeout(HiveWriter.java:337) ~[StormTopology-0.1.jar:?]
... 15 more
Caused by: java.lang.NoSuchFieldError: METASTORE_FILTER_HOOK
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.loadFilterHooks(HiveMetaStoreClient.java:240) ~[StormTopology-0.1.jar:?]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:192) ~[StormTopology-0.1.jar:?]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) ~[StormTopology-0.1.jar:?]
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448) ~[StormTopology-0.1.jar:?]
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274) ~[StormTopology-0.1.jar:?]
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) ~[StormTopology-0.1.jar:?]
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) ~[StormTopology-0.1.jar:?]
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) ~[StormTopology-0.1.jar:?]
at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:229) ~[StormTopology-0.1.jar:?]
at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:226) ~[StormTopology-0.1.jar:?]
at org.apache.storm.hive.common.HiveWriter$9.call(HiveWriter.java:332) ~[StormTopology-0.1.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_71]
... 1 more
2) Using Hive library version 2.0.0 (http://search.maven.org/#artifactdetails|org.apache.hive|hive|2.0.0|pom) and configuration as in the attached pom2.xml file, the error returned is:
32028 [Thread-12-hiveBolt] ERROR b.s.d.executor -
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.conf.HiveConf
at org.apache.storm.hive.common.HiveWriter.callWithTimeout(HiveWriter.java:357) ~[storm-hive-0.10.0.jar:0.10.0]
at org.apache.storm.hive.common.HiveWriter.newConnection(HiveWriter.java:226) ~[storm-hive-0.10.0.jar:0.10.0]
at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:69) ~[storm-hive-0.10.0.jar:0.10.0]
at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[storm-hive-0.10.0.jar:0.10.0]
at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:219) ~[storm-hive-0.10.0.jar:0.10.0]
at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:102) [storm-hive-0.10.0.jar:0.10.0]
at backtype.storm.daemon.executor$fn__5694$tuple_action_fn__5696.invoke(executor.clj:690) [storm-core-0.10.0.jar:0.10.0]
at backtype.storm.daemon.executor$mk_task_receiver$fn__5615.invoke(executor.clj:436) [storm-core-0.10.0.jar:0.10.0]
at backtype.storm.disruptor$clojure_handler$reify__5189.onEvent(disruptor.clj:58) [storm-core-0.10.0.jar:0.10.0]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:132) [storm-core-0.10.0.jar:0.10.0]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:106) [storm-core-0.10.0.jar:0.10.0]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [storm-core-0.10.0.jar:0.10.0]
at backtype.storm.daemon.executor$fn__5694$fn__5707$fn__5758.invoke(executor.clj:819) [storm-core-0.10.0.jar:0.10.0]
at backtype.storm.util$async_loop$fn__545.invoke(util.clj:479) [storm-core-0.10.0.jar:0.10.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.conf.HiveConf
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_31]
at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_31]
at org.apache.storm.hive.common.HiveWriter.callWithTimeout(HiveWriter.java:337) ~[storm-hive-0.10.0.jar:0.10.0]
... 15 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.conf.HiveConf
at org.apache.hive.hcatalog.streaming.HiveEndPoint.createHiveConf(HiveEndPoint.java:842) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:268) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:229) ~[storm-hive-0.10.0.jar:0.10.0]
at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:226) ~[storm-hive-0.10.0.jar:0.10.0]
at org.apache.storm.hive.common.HiveWriter$9.call(HiveWriter.java:332) ~[storm-hive-0.10.0.jar:0.10.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_31]
... 1 more
32029 [Thread-14-__acker] INFO b.s.d.executor - BOLT ack TASK: 1 TIME: TUPLE: source: parserBolt:5, stream: __ack_ack, id: {}, [820336490148731685 6454746331808199053]
32029 [Thread-14-__acker] INFO b.s.d.executor - Execute done TUPLE source
Also, we included external configuration files in project (hive-site.xml and hive-env.sh) as indicated in hortonworks guidelines. This is the hive’s bolt code:
private void createHiveBolt(TopologyBuilder builder)
{
try
{
// Record Writer configuration
DelimitedRecordHiveMapper mapper = new DelimitedRecordHiveMapper()
.withColumnFields(DataScheme.GetHiveFields());
HiveOptions hiveOptions;
hiveOptions = new HiveOptions(topologyConf.HiveMetastore, topologyConf.HiveDbName, topologyConf.HiveTableName, mapper)
.withTxnsPerBatch(2)
.withBatchSize(100)
.withIdleTimeout(10);
builder.setBolt(HIVE_BOLT_ID, new HiveBolt(hiveOptions), topologyConf.ParallelHint).shuffleGrouping(PARSER_BOLT_ID);
}
catch(Exception ex)
{
logger.error(ex.getMessage());
}
}
How we can solve this issues?
Thank you
