Hive kafka integration giving error class not found exception : message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found

yukti — Tue, 03 Sep 2019 02:36:13 GMT

Hi Team,
I was trying to read data from kafka topics by using hive table.

SInce my data is in avro format so trying to create a hive using below statement:

CREATE EXTERNAL TABLE kafka_t_avro
   (`timestamp` timestamp , `page` string, `newPage` boolean,
   added int, deleted bigint, delta double)
   STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
   TBLPROPERTIES
   ("kafka.topic" = "topic-name",
   "kafka.bootstrap.servers"="hostname:9092",
   "kafka.serde.class"="org.apache.hadoop.hive.serde2.avro.AvroSerDe");
It is giving me below error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found).

I have added the below jars in my class path:

kafka-handler-3.1.0.3.1.0.0-78.jar;
hive-serde-0.10.0.jar;
hive-exec-3.1.0.3.1.0.0-78.jar;

@jsensharma

Re: Hive kafka integration giving error class not found exception : message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found

jsensharma — Tue, 03 Sep 2019 08:05:41 GMT

@yukti

As you have already got the 3.1.0.0-78 version of JARs so for the missing class "org.apache.hadoop.hive.kafka.KafkaSerDe" can you try using the following JAR (instead of "hive-serde-0.10.0.jar'")?

In My HDP 3 installation on hive host i can see it here " usr/hdp/current/hive-server2/lib/kafka-handler-3.1.0.3.1.0.0-78.jar"

# /usr/jdk64/jdk1.8.0_112/bin/javap -cp /usr/hdp/current/hive-server2/lib/kafka-handler-3.1.0.3.1.0.0-78.jar org.apache.hadoop.hive.kafka.KafkaSerDe Compiled from "KafkaSerDe.java" public class org.apache.hadoop.hive.kafka.KafkaSerDe extends org.apache.hadoop.hive.serde2.AbstractSerDe { public org.apache.hadoop.hive.kafka.KafkaSerDe(); public void initialize(org.apache.hadoop.conf.Configuration, java.util.Properties) throws org.apache.hadoop.hive.serde2.SerDeException; public java.lang.Class<? extends org.apache.hadoop.io.Writable> getSerializedClass(); public org.apache.hadoop.io.Writable serialize(java.lang.Object, org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector) throws org.apache.hadoop.hive.serde2.SerDeException; public org.apache.hadoop.hive.serde2.SerDeStats getSerDeStats(); public java.lang.Object deserialize(org.apache.hadoop.io.Writable) throws org.apache.hadoop.hive.serde2.SerDeException; java.util.ArrayList<java.lang.Object> deserializeKWritable(org.apache.hadoop.hive.kafka.KafkaWritable) throws org.apache.hadoop.hive.serde2.SerDeException; public org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector getObjectInspector(); static {}; }

question Hive kafka integration giving error class not found exception : message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found in Support Questions

Hive kafka integration giving error class not found exception : message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found

Re: Hive kafka integration giving error class not found exception : message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.kafka.KafkaSerDe not found