question Re: Creating hive table to read data from kafka topics giving error "Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)" in Support Questions

Creating hive table to read data from kafka topics giving error "Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)"

yukti — Mon, 02 Sep 2019 03:36:53 GMT

I am trying to create hive table to read data from kafka topics.

I am using CDH 6.2.0.
I am adding the below jar before creating the table :

kafka-handler-3.1.0.3.1.0.0-78.jar;
hive-serde-0.10.0.jar;
hive-metastore-0.9.0.jar;

below is the create table statement:

CREATE EXTERNAL TABLE kafka_table
(`timestamp` timestamp , `page` string, `newPage` boolean,
added int, deleted bigint, delta double)
STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
TBLPROPERTIES
("kafka.topic" = "topic-name", "kafka.bootstrap.servers"="hostname:9092");

giving me below error:

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.DefaultHiveMetaHook
org/apache/hadoop/hive/metastore/DefaultHiveMetaHook

Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)

Re: Creating hive table to read data from kafka topics giving error "Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)"

jsensharma — Mon, 02 Sep 2019 03:54:12 GMT

@yukti

You seems to be using OLD "hive-metastore-0.9.0.jar" JAR file. When you are using it with "kafka-handler-3.1.0.3.1.0.0-78.jar" jar.

Do you see the mentioned class inside your JAR?

# javap -cp /PATH/TO/hive-metastore-0.9.0.jar org.apache.hadoop.hive.metastore.DefaultHiveMetaHook

As i see your Kafka JAR version as "kafka-handler-3.1.0.3.1.0.0-78.jar" Which has a Dependency to . hive-exec module of the same version i guess. Just try to look at the "kafka-handler-3.1.0.3.1.0.0-78.jar"/META-INF/maven/org.apache.hive/kafka-handler/pom.xml" . file and then you will see that it needs the same verison of hive-exec

# grep 'hive-exec' -A2 META-INF/maven/org.apache.hive/kafka-handler/pom.xml <artifactId>hive-exec</artifactId> <scope>provided</scope> <version>${project.version}</version>

So i guess you should be using the following JAR (if you are using HDP installation)

/usr/hdp/current/hive-metastore/lib/hive-standalone-metastore-3.1.0.3.1.0.0-78.jar OR /usr/hdp/current/hive-metastore/lib/hive-exec-3.1.0.3.1.0.0-78.jar

Re: Creating hive table to read data from kafka topics giving error "Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)"

jsensharma — Mon, 02 Sep 2019 05:15:26 GMT

@yukti

You seems to be using incorrect version of "hive-exec" / "hive-metastore" JAR.

Can you please tell me from where did you get the 3.1.0.;0-78 version of JARs (as your Kafka jar version is"kafka-handler-3.1.0.3.1.0.0-78.jar" so looks like you might have downloaded that JAR from HDP 3.1 installation? Is that correct? If yes then can you also take the hive-metastore JAR of the same version and then try.

You can see the difference here between "hive-metastore-3.1.0.3.1.0.0-78.jar" and "hive-metastore-0.9.0.jar" JARs. From my HDP 3.1 installation.

# ls -l /usr/hdp/3.1.0.0-78/hive/lib/hive-metastore.jar lrwxrwxrwx. 1 root root 35 Feb 22 2019 /usr/hdp/3.1.0.0-78/hive/lib/hive-metastore.jar -> hive-metastore-3.1.0.3.1.0.0-78.jar # /usr/jdk64/jdk1.8.0_112/bin/javap -cp /usr/hdp/current/hive-metastore/lib/hive-standalone-metastore-3.1.0.3.1.0.0-78.jar org.apache.hadoop.hive.metastore.DefaultHiveMetaHook Compiled from "DefaultHiveMetaHook.java" public abstract class org.apache.hadoop.hive.metastore.DefaultHiveMetaHook implements org.apache.hadoop.hive.metastore.HiveMetaHook { public org.apache.hadoop.hive.metastore.DefaultHiveMetaHook(); public abstract void commitInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException; public abstract void preInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException; public abstract void rollbackInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException; }

On the other have we can see that "hive-metastore-0.9.0.jar" jar does not contains that class.

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /tmp/hive-metastore-0.9.0.jar org.apache.hadoop.hive.metastore.DefaultHiveMetaHook Error: class not found: org.apache.hadoop.hive.metastore.DefaultHiveMetaHook

If your question is answered then, Please make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Re: Creating hive table to read data from kafka topics giving error "Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)"

yukti — Mon, 02 Sep 2019 10:22:31 GMT

Hi jay,

That issue get resolved but then I tried altering table it was giving me error:

hive> ALTER TABLE kafka_table SET TBLPROPERTIES ("kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe");
FAILED: SemanticException [Error 10134]: ALTER TABLE cannot be used for a non-native table kafka_table

So I dropped that table ad try creating a new one having below statements:

CREATE EXTERNAL TABLE kafka_t_avro
   (`timestamp` timestamp , `page` string, `newPage` boolean,
   added int, deleted bigint, delta double)
   STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
   TBLPROPERTIES
   ("kafka.topic" = "topic-name",
   "kafka.bootstrap.servers"="hostname:9092",
   "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe");

Itsi sgiving me below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found)

Re: Creating hive table to read data from kafka topics giving error "Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)"

jsensharma — Mon, 02 Sep 2019 10:25:51 GMT

@yukti

Good to know that your originally reported issue is resolved.

If your question is answered then, Please make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

For the new error, It will be best of you open a separate Topic/Thread that way user of this thread will not get confused with different errors. The new error also looks like related to incorrect JAR version.

Re: Creating hive table to read data from kafka topics giving error "Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)"

yukti — Tue, 03 Sep 2019 02:29:21 GMT

Thanks jay!!!

I am starting a new thread.

Re: Creating hive table to read data from kafka topics giving error "Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)"

AmroSaleh — Thu, 23 Apr 2020 19:19:28 GMT

Hello Friend ,

ow can i add the kafka-handler.jar in cloudera cluster?? from the manager or i have to copy the jar t specific location? can you explain