Skip to main content
yukti1
yukti1
Gold ⭐️
September 3, 2019
Solved

Hive kafka integration giving error class not found exception : message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found

Hi Team,
I was trying to read data from kafka topics by using hive table.

SInce my data is in avro format so trying to create a hive using below statement:


CREATE EXTERNAL TABLE kafka_t_avro
   (`timestamp` timestamp , `page` string,  `newPage` boolean,
   added int, deleted bigint, delta double)
   STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
   TBLPROPERTIES
   ("kafka.topic" = "topic-name",
   "kafka.bootstrap.servers"="hostname:9092",
   "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe");
It is giving me below error:
 
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found).

 

I have added the below jars in my class path:

 

kafka-handler-3.1.0.3.1.0.0-78.jar;
hive-serde-0.10.0.jar;
hive-exec-3.1.0.3.1.0.0-78.jar;

@jsensharma 

jsensharma
jsensharmaAnswer
Platinum ⭐️⭐️
September 3, 2019

@yukti1 

 

As you have already got the 3.1.0.0-78 version of JARs so for the missing class "org.apache.hadoop.hive.kafka.KafkaSerDe" can you try using the following JAR (instead of "hive-serde-0.10.0.jar'")?

 

In My HDP 3 installation on hive host i can see it here " usr/hdp/current/hive-server2/lib/kafka-handler-3.1.0.3.1.0.0-78.jar"

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /usr/hdp/current/hive-server2/lib/kafka-handler-3.1.0.3.1.0.0-78.jar org.apache.hadoop.hive.kafka.KafkaSerDe

Compiled from "KafkaSerDe.java"
public class org.apache.hadoop.hive.kafka.KafkaSerDe extends org.apache.hadoop.hive.serde2.AbstractSerDe {
  public org.apache.hadoop.hive.kafka.KafkaSerDe();
  public void initialize(org.apache.hadoop.conf.Configuration, java.util.Properties) throws org.apache.hadoop.hive.serde2.SerDeException;
  public java.lang.Class<? extends org.apache.hadoop.io.Writable> getSerializedClass();
  public org.apache.hadoop.io.Writable serialize(java.lang.Object, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector) throws org.apache.hadoop.hive.serde2.SerDeException;
  public org.apache.hadoop.hive.serde2.SerDeStats getSerDeStats();
  public java.lang.Object deserialize(org.apache.hadoop.io.Writable) throws org.apache.hadoop.hive.serde2.SerDeException;
  java.util.ArrayList<java.lang.Object> deserializeKWritable(org.apache.hadoop.hive.kafka.KafkaWritable) throws org.apache.hadoop.hive.serde2.SerDeException;
  public org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector getObjectInspector();
  static {};
}

 

 

View in thread

1 reply

jsensharma
Platinum ⭐️⭐️
jsensharmaAnswer
Platinum ⭐️⭐️
September 3, 2019

@yukti1 

 

As you have already got the 3.1.0.0-78 version of JARs so for the missing class "org.apache.hadoop.hive.kafka.KafkaSerDe" can you try using the following JAR (instead of "hive-serde-0.10.0.jar'")?

 

In My HDP 3 installation on hive host i can see it here " usr/hdp/current/hive-server2/lib/kafka-handler-3.1.0.3.1.0.0-78.jar"

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /usr/hdp/current/hive-server2/lib/kafka-handler-3.1.0.3.1.0.0-78.jar org.apache.hadoop.hive.kafka.KafkaSerDe

Compiled from "KafkaSerDe.java"
public class org.apache.hadoop.hive.kafka.KafkaSerDe extends org.apache.hadoop.hive.serde2.AbstractSerDe {
  public org.apache.hadoop.hive.kafka.KafkaSerDe();
  public void initialize(org.apache.hadoop.conf.Configuration, java.util.Properties) throws org.apache.hadoop.hive.serde2.SerDeException;
  public java.lang.Class<? extends org.apache.hadoop.io.Writable> getSerializedClass();
  public org.apache.hadoop.io.Writable serialize(java.lang.Object, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector) throws org.apache.hadoop.hive.serde2.SerDeException;
  public org.apache.hadoop.hive.serde2.SerDeStats getSerDeStats();
  public java.lang.Object deserialize(org.apache.hadoop.io.Writable) throws org.apache.hadoop.hive.serde2.SerDeException;
  java.util.ArrayList<java.lang.Object> deserializeKWritable(org.apache.hadoop.hive.kafka.KafkaWritable) throws org.apache.hadoop.hive.serde2.SerDeException;
  public org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector getObjectInspector();
  static {};
}

 

 

Translate
Terms & ConditionsAccessibility statement