Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Horton Avro decoder to consume Kafka messages with Spark and dynamically discover schema in schema registry

Highlighted

Horton Avro decoder to consume Kafka messages with Spark and dynamically discover schema in schema registry

New Contributor

Is there any Horton Avro decoder similar to the Confluent KafkaAvroDecoder that handles deserialization and at the same time dynamic schema discovery by accessing schema registry trough a rest service client.

I need to consume with Spark some Kafka messages serialized in Avro format while the schema is published in Horton schema registry. The decoder needs to deserialize but also contact schema registry to retrieve the schema.

Confluent offers this out of the box.

KafkaAvroDecoder would be the "valueDecoderClass" parameter for the following API:

https://spark.apache.org/docs/1.6.2/api/java/org/apache/spark/streaming/kafka/KafkaUtils.html#create...

If I use some standard decoder like StringDecoder I can consume the messages but in serialized format:

44310380200008480�����W�����W-122.05116637.359015

Don't have an account?
Coming from Hortonworks? Activate your account here