Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Use my own serde with KafkaStorageHandler

Use my own serde with KafkaStorageHandler

Expert Contributor

I want to use the new KafkaStorageHandler, which looks awesome. The only thing is that the avro in kafka is not standard (I am looking at you, confluent), I thus need to use my own serde (well, the serde from confluent).

I added wihout error to hive the relevant jar which contains io.confluent.kafka.streams.serdes.avro.GenericAvroSerde.

add jar hdfs:///tmp/kafka-streams-avro-serde-5.1.0.jar; 

If I now try to create the external table :

CREATE EXTERNAL TABLE click ( ... )
STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler'
  TBLPROPERTIES (
    "kafka.topic" = "click", "kafka.bootstrap.servers"="kafka:9092"
    ,"kafka.serde.class"="io.confluent.kafka.streams.serdes.avro.GenericAvroSerde"
);

Hive bails out and says:

Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: java.lang.ClassNotFoundException: io.confluent.kafka.streams.serdes.avro.GenericAvroSerde (state=08S01,code=1)

How can I tell hive where to find this class?

Thanks,

Don't have an account?
Coming from Hortonworks? Activate your account here