Creating hive table to read data from kafka topics giving error "Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)"

[#1]

I am trying to create hive table to read data from kafka topics.

I am using CDH 6.2.0.
I am adding the below jar before creating the table :

kafka-handler-3.1.0.3.1.0.0-78.jar;
hive-serde-0.10.0.jar;
hive-metastore-0.9.0.jar;

below is the create table statement:

CREATE EXTERNAL TABLE kafka_table
  (`timestamp` timestamp , `page` string,  `newPage` boolean,
  added int, deleted bigint, delta double)
  STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
  TBLPROPERTIES
  ("kafka.topic" = "topic-name", "kafka.bootstrap.servers"="hostname:9092");

giving me below error:

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.DefaultHiveMetaHook
org/apache/hadoop/hive/metastore/DefaultHiveMetaHook

Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)

[#2 · Accepted solution]

@yukti 

You seems to be using OLD  "hive-metastore-0.9.0.jar" JAR file. When you are using it with "kafka-handler-3.1.0.3.1.0.0-78.jar" jar.

Do you see the mentioned class inside your JAR?

# javap -cp /PATH/TO/hive-metastore-0.9.0.jar  org.apache.hadoop.hive.metastore.DefaultHiveMetaHook

As i see your Kafka JAR version as "kafka-handler-3.1.0.3.1.0.0-78.jar"  Which has a Dependency to . hive-exec module of the same version i guess.  Just try to look at the "kafka-handler-3.1.0.3.1.0.0-78.jar"/META-INF/maven/org.apache.hive/kafka-handler/pom.xml" . file and then you will see that it needs the same verison of hive-exec

# grep 'hive-exec' -A2 META-INF/maven/org.apache.hive/kafka-handler/pom.xml 
      <artifactId>hive-exec</artifactId>
      <scope>provided</scope>
      <version>${project.version}</version>

So i guess you should be using the following JAR (if you are using HDP installation)

/usr/hdp/current/hive-metastore/lib/hive-standalone-metastore-3.1.0.3.1.0.0-78.jar
OR
/usr/hdp/current/hive-metastore/lib/hive-exec-3.1.0.3.1.0.0-78.jar

[#3 · Accepted solution]

@yukti 

You seems to be using incorrect version of "hive-exec" / "hive-metastore" JAR.

Can you please tell me from where did you get the 3.1.0.;0-78 version of JARs (as your Kafka jar version is"kafka-handler-3.1.0.3.1.0.0-78.jar" so looks like you might have downloaded that JAR from HDP 3.1 installation?  Is that correct?  If yes then can you also take the hive-metastore JAR of the same version and then try.

You can see the difference here between "hive-metastore-3.1.0.3.1.0.0-78.jar" and "hive-metastore-0.9.0.jar" JARs. From my HDP 3.1 installation.

# ls -l /usr/hdp/3.1.0.0-78/hive/lib/hive-metastore.jar
lrwxrwxrwx. 1 root root 35 Feb 22  2019 /usr/hdp/3.1.0.0-78/hive/lib/hive-metastore.jar -> hive-metastore-3.1.0.3.1.0.0-78.jar

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /usr/hdp/current/hive-metastore/lib/hive-standalone-metastore-3.1.0.3.1.0.0-78.jar org.apache.hadoop.hive.metastore.DefaultHiveMetaHook
Compiled from "DefaultHiveMetaHook.java"
public abstract class org.apache.hadoop.hive.metastore.DefaultHiveMetaHook implements org.apache.hadoop.hive.metastore.HiveMetaHook {
  public org.apache.hadoop.hive.metastore.DefaultHiveMetaHook();
  public abstract void commitInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;
  public abstract void preInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;
  public abstract void rollbackInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;
}

.

On the other have we can see that "hive-metastore-0.9.0.jar" jar does not contains that class.

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /tmp/hive-metastore-0.9.0.jar org.apache.hadoop.hive.metastore.DefaultHiveMetaHook
Error: class not found: org.apache.hadoop.hive.metastore.DefaultHiveMetaHook

.

If your question is answered then, Please make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#4 · Accepted solution]

@yukti 

You seems to be using OLD  "hive-metastore-0.9.0.jar" JAR file. When you are using it with "kafka-handler-3.1.0.3.1.0.0-78.jar" jar.

Do you see the mentioned class inside your JAR?

# javap -cp /PATH/TO/hive-metastore-0.9.0.jar  org.apache.hadoop.hive.metastore.DefaultHiveMetaHook

As i see your Kafka JAR version as "kafka-handler-3.1.0.3.1.0.0-78.jar"  Which has a Dependency to . hive-exec module of the same version i guess.  Just try to look at the "kafka-handler-3.1.0.3.1.0.0-78.jar"/META-INF/maven/org.apache.hive/kafka-handler/pom.xml" . file and then you will see that it needs the same verison of hive-exec

# grep 'hive-exec' -A2 META-INF/maven/org.apache.hive/kafka-handler/pom.xml 
      <artifactId>hive-exec</artifactId>
      <scope>provided</scope>
      <version>${project.version}</version>

So i guess you should be using the following JAR (if you are using HDP installation)

/usr/hdp/current/hive-metastore/lib/hive-standalone-metastore-3.1.0.3.1.0.0-78.jar
OR
/usr/hdp/current/hive-metastore/lib/hive-exec-3.1.0.3.1.0.0-78.jar

[#5 · Accepted solution]

@yukti 

You seems to be using incorrect version of "hive-exec" / "hive-metastore" JAR.

Can you please tell me from where did you get the 3.1.0.;0-78 version of JARs (as your Kafka jar version is"kafka-handler-3.1.0.3.1.0.0-78.jar" so looks like you might have downloaded that JAR from HDP 3.1 installation?  Is that correct?  If yes then can you also take the hive-metastore JAR of the same version and then try.

You can see the difference here between "hive-metastore-3.1.0.3.1.0.0-78.jar" and "hive-metastore-0.9.0.jar" JARs. From my HDP 3.1 installation.

# ls -l /usr/hdp/3.1.0.0-78/hive/lib/hive-metastore.jar
lrwxrwxrwx. 1 root root 35 Feb 22  2019 /usr/hdp/3.1.0.0-78/hive/lib/hive-metastore.jar -> hive-metastore-3.1.0.3.1.0.0-78.jar

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /usr/hdp/current/hive-metastore/lib/hive-standalone-metastore-3.1.0.3.1.0.0-78.jar org.apache.hadoop.hive.metastore.DefaultHiveMetaHook
Compiled from "DefaultHiveMetaHook.java"
public abstract class org.apache.hadoop.hive.metastore.DefaultHiveMetaHook implements org.apache.hadoop.hive.metastore.HiveMetaHook {
  public org.apache.hadoop.hive.metastore.DefaultHiveMetaHook();
  public abstract void commitInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;
  public abstract void preInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;
  public abstract void rollbackInsertTable(org.apache.hadoop.hive.metastore.api.Table, boolean) throws org.apache.hadoop.hive.metastore.api.MetaException;
}

.

On the other have we can see that "hive-metastore-0.9.0.jar" jar does not contains that class.

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /tmp/hive-metastore-0.9.0.jar org.apache.hadoop.hive.metastore.DefaultHiveMetaHook
Error: class not found: org.apache.hadoop.hive.metastore.DefaultHiveMetaHook

.

If your question is answered then, Please make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#6]

Hi jay,

That issue get resolved but then I tried altering table it was giving me error:

hive> ALTER TABLE kafka_table SET TBLPROPERTIES ("kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe");
FAILED: SemanticException [Error 10134]: ALTER TABLE cannot be used for a non-native table kafka_table

So I dropped that table ad try creating a new one having below statements:

CREATE EXTERNAL TABLE kafka_t_avro
   (`timestamp` timestamp , `page` string,  `newPage` boolean,
   added int, deleted bigint, delta double)
   STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
   TBLPROPERTIES
   ("kafka.topic" = "topic-name",
   "kafka.bootstrap.servers"="hostname:9092",
   "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe");

Itsi sgiving me below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found)

[#7 · Accepted solution]

@yukti 

Good to know that your originally reported issue is resolved.

If your question is answered then, Please make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

.

.

For the new error, It will be best of you open a separate Topic/Thread that way user of this thread will not get confused with different errors.   The new error also looks like related to incorrect JAR version.

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found)

[#8]

Thanks jay!!!

I am starting a new thread.

[#9]

Hello Friend ,

ow can i add the kafka-handler.jar  in cloudera cluster?? from the manager or i have to copy the jar t specific location? can you explain