Error in executing Hive Bolt with Storm

vergari — Wed, 09 Mar 2016 19:56:07 GMT

Hi all,

we are developing a storm topology to write streaming data in hive database but the following errors occurs during executions: 1) Using Hive library version 1.2.1 (http://search.maven.org/#artifactdetails|org.apache.hive|hive|1.2.1|pom) and configuration as in the attached pom1.xml file, the error is:

43088 [Thread-12-hiveBolt] ERROR b.s.d.executor -
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoSuchFieldError: METASTORE_FILTER_HOOK
        at org.apache.storm.hive.common.HiveWriter.callWithTimeout(HiveWriter.java:357) ~[StormTopology-0.1.jar:?]
        at org.apache.storm.hive.common.HiveWriter.newConnection(HiveWriter.java:226) ~[StormTopology-0.1.jar:?]
        at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:69) ~[StormTopology-0.1.jar:?]
        at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[StormTopology-0.1.jar:?]
        at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:219) ~[StormTopology-0.1.jar:?]
        at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:102) [StormTopology-0.1.jar:?]
        at backtype.storm.daemon.executor$fn__5694$tuple_action_fn__5696.invoke(executor.clj:690) [StormTopology-0.1.jar:?]
        at backtype.storm.daemon.executor$mk_task_receiver$fn__5615.invoke(executor.clj:436) [StormTopology-0.1.jar:?]
        at backtype.storm.disruptor$clojure_handler$reify__5189.onEvent(disruptor.clj:58) [StormTopology-0.1.jar:?]
        at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:132) [StormTopology-0.1.jar:?]
        at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:106) [StormTopology-0.1.jar:?]
        at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [StormTopology-0.1.jar:?]
        at backtype.storm.daemon.executor$fn__5694$fn__5707$fn__5758.invoke(executor.clj:819) [StormTopology-0.1.jar:?]
        at backtype.storm.util$async_loop$fn__545.invoke(util.clj:479) [StormTopology-0.1.jar:?]
        at clojure.lang.AFn.run(AFn.java:22) [StormTopology-0.1.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchFieldError: METASTORE_FILTER_HOOK
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_71]
        at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_71]
        at org.apache.storm.hive.common.HiveWriter.callWithTimeout(HiveWriter.java:337) ~[StormTopology-0.1.jar:?]
        ... 15 more
Caused by: java.lang.NoSuchFieldError: METASTORE_FILTER_HOOK
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.loadFilterHooks(HiveMetaStoreClient.java:240) ~[StormTopology-0.1.jar:?]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:192) ~[StormTopology-0.1.jar:?]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181) ~[StormTopology-0.1.jar:?]
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448) ~[StormTopology-0.1.jar:?]
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274) ~[StormTopology-0.1.jar:?]
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) ~[StormTopology-0.1.jar:?]
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) ~[StormTopology-0.1.jar:?]
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) ~[StormTopology-0.1.jar:?]
        at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:229) ~[StormTopology-0.1.jar:?]
        at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:226) ~[StormTopology-0.1.jar:?]
        at org.apache.storm.hive.common.HiveWriter$9.call(HiveWriter.java:332) ~[StormTopology-0.1.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_71]
        ... 1 more

2) Using Hive library version 2.0.0 (http://search.maven.org/#artifactdetails|org.apache.hive|hive|2.0.0|pom) and configuration as in the attached pom2.xml file, the error returned is:

32028 [Thread-12-hiveBolt] ERROR b.s.d.executor - 
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.conf.HiveConf
   at org.apache.storm.hive.common.HiveWriter.callWithTimeout(HiveWriter.java:357) ~[storm-hive-0.10.0.jar:0.10.0]
   at org.apache.storm.hive.common.HiveWriter.newConnection(HiveWriter.java:226) ~[storm-hive-0.10.0.jar:0.10.0]
   at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:69) ~[storm-hive-0.10.0.jar:0.10.0]
   at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:45) ~[storm-hive-0.10.0.jar:0.10.0]
   at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:219) ~[storm-hive-0.10.0.jar:0.10.0]
   at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:102) [storm-hive-0.10.0.jar:0.10.0]
   at backtype.storm.daemon.executor$fn__5694$tuple_action_fn__5696.invoke(executor.clj:690) [storm-core-0.10.0.jar:0.10.0]
   at backtype.storm.daemon.executor$mk_task_receiver$fn__5615.invoke(executor.clj:436) [storm-core-0.10.0.jar:0.10.0]
   at backtype.storm.disruptor$clojure_handler$reify__5189.onEvent(disruptor.clj:58) [storm-core-0.10.0.jar:0.10.0]
   at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:132) [storm-core-0.10.0.jar:0.10.0]
   at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:106) [storm-core-0.10.0.jar:0.10.0]
   at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [storm-core-0.10.0.jar:0.10.0]
   at backtype.storm.daemon.executor$fn__5694$fn__5707$fn__5758.invoke(executor.clj:819) [storm-core-0.10.0.jar:0.10.0]
   at backtype.storm.util$async_loop$fn__545.invoke(util.clj:479) [storm-core-0.10.0.jar:0.10.0]
   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_31]
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.conf.HiveConf
   at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_31]
   at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_31]
   at org.apache.storm.hive.common.HiveWriter.callWithTimeout(HiveWriter.java:337) ~[storm-hive-0.10.0.jar:0.10.0]
   ... 15 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.conf.HiveConf
   at org.apache.hive.hcatalog.streaming.HiveEndPoint.createHiveConf(HiveEndPoint.java:842) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
   at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:268) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
   at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
   at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
   at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]
   at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:229) ~[storm-hive-0.10.0.jar:0.10.0]
   at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:226) ~[storm-hive-0.10.0.jar:0.10.0]
   at org.apache.storm.hive.common.HiveWriter$9.call(HiveWriter.java:332) ~[storm-hive-0.10.0.jar:0.10.0]
   at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_31]
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_31]
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_31]
   ... 1 more
32029 [Thread-14-__acker] INFO  b.s.d.executor - BOLT ack TASK: 1 TIME:  TUPLE: source: parserBolt:5, stream: __ack_ack, id: {}, [820336490148731685 6454746331808199053]
32029 [Thread-14-__acker] INFO  b.s.d.executor - Execute done TUPLE source

Also, we included external configuration files in project (hive-site.xml and hive-env.sh) as indicated in hortonworks guidelines. This is the hive’s bolt code:

  private void createHiveBolt(TopologyBuilder builder)
    {
        try
        {             
            // Record Writer configuration
            DelimitedRecordHiveMapper mapper = new DelimitedRecordHiveMapper()
                .withColumnFields(DataScheme.GetHiveFields());
            
            HiveOptions hiveOptions;
            
            hiveOptions = new HiveOptions(topologyConf.HiveMetastore, topologyConf.HiveDbName, topologyConf.HiveTableName, mapper)
                .withTxnsPerBatch(2)
                .withBatchSize(100)
                .withIdleTimeout(10);
            
            
            builder.setBolt(HIVE_BOLT_ID, new HiveBolt(hiveOptions), topologyConf.ParallelHint).shuffleGrouping(PARSER_BOLT_ID);            
            
        }
        catch(Exception ex)
        {
            logger.error(ex.getMessage());            
        }
    }

How we can solve this issues?

Thank you

Re: Error in executing Hive Bolt with Storm

aervits — Wed, 09 Mar 2016 20:53:28 GMT

Please use this guide to package your topology, do no use hive 2.0 http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_storm-user-guide/content/ch_storm-pkg-topologies.html read the rest of the guide as it has storm-hive examples

Re: Error in executing Hive Bolt with Storm

vergari — Thu, 10 Mar 2016 18:30:14 GMT

We used the guides you posted and now all works right! Thank you!

question Error in executing Hive Bolt with Storm in Archives of Support Questions (Read Only)

Error in executing Hive Bolt with Storm

Re: Error in executing Hive Bolt with Storm

Re: Error in executing Hive Bolt with Storm