Support Questions
Find answers, ask questions, and share your expertise

Use my own serde with KafkaStorageHandler

Use my own serde with KafkaStorageHandler

Expert Contributor

I want to use the new KafkaStorageHandler, which looks awesome. The only thing is that the avro in kafka is not standard (I am looking at you, confluent), I thus need to use my own serde (well, the serde from confluent).

I added wihout error to hive the relevant jar which contains io.confluent.kafka.streams.serdes.avro.GenericAvroSerde.

add jar hdfs:///tmp/kafka-streams-avro-serde-5.1.0.jar; 

If I now try to create the external table :

CREATE EXTERNAL TABLE click ( ... )
STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
  TBLPROPERTIES (
    "kafka.topic" = "click", "kafka.bootstrap.servers"="kafka:9092"
    ,"kafka.serde.class"="io.confluent.kafka.streams.serdes.avro.GenericAvroSerde"
);

Hive bails out and says:

Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: java.lang.ClassNotFoundException: io.confluent.kafka.streams.serdes.avro.GenericAvroSerde (state=08S01,code=1)

How can I tell hive where to find this class?

Thanks,