Hive kafka integration giving error class not found exception : message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found

[#1]

Hi Team,
I was trying to read data from kafka topics by using hive table.

SInce my data is in avro format so trying to create a hive using below statement:

CREATE EXTERNAL TABLE kafka_t_avro
   (`timestamp` timestamp , `page` string,  `newPage` boolean,
   added int, deleted bigint, delta double)
   STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
   TBLPROPERTIES
   ("kafka.topic" = "topic-name",
   "kafka.bootstrap.servers"="hostname:9092",
   "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe");
It is giving me below error:
 
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found).

I have added the below jars in my class path:

kafka-handler-3.1.0.3.1.0.0-78.jar;
hive-serde-0.10.0.jar;
hive-exec-3.1.0.3.1.0.0-78.jar;

@jsensharma 

[#2 · Accepted solution]

@yukti 

As you have already got the 3.1.0.0-78 version of JARs so for the missing class "org.apache.hadoop.hive.kafka.KafkaSerDe" can you try using the following JAR (instead of "hive-serde-0.10.0.jar'")?

In My HDP 3 installation on hive host i can see it here " usr/hdp/current/hive-server2/lib/kafka-handler-3.1.0.3.1.0.0-78.jar"

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /usr/hdp/current/hive-server2/lib/kafka-handler-3.1.0.3.1.0.0-78.jar org.apache.hadoop.hive.kafka.KafkaSerDe

Compiled from "KafkaSerDe.java"
public class org.apache.hadoop.hive.kafka.KafkaSerDe extends org.apache.hadoop.hive.serde2.AbstractSerDe {
  public org.apache.hadoop.hive.kafka.KafkaSerDe();
  public void initialize(org.apache.hadoop.conf.Configuration, java.util.Properties) throws org.apache.hadoop.hive.serde2.SerDeException;
  public java.lang.Class<? extends org.apache.hadoop.io.Writable> getSerializedClass();
  public org.apache.hadoop.io.Writable serialize(java.lang.Object, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector) throws org.apache.hadoop.hive.serde2.SerDeException;
  public org.apache.hadoop.hive.serde2.SerDeStats getSerDeStats();
  public java.lang.Object deserialize(org.apache.hadoop.io.Writable) throws org.apache.hadoop.hive.serde2.SerDeException;
  java.util.ArrayList<java.lang.Object> deserializeKWritable(org.apache.hadoop.hive.kafka.KafkaWritable) throws org.apache.hadoop.hive.serde2.SerDeException;
  public org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector getObjectInspector();
  static {};
}