Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

How do i read kafka avro messages in spark streaming?

New Contributor

Hi,

Does anyone know how to read the avro messages from kafka in spark streaming? Kindly comment or PM me.

Regards, Sathwik

1 REPLY 1

Expert Contributor

Your question leads me believe that you know how to write avro messages into a Kafka topic. Once your avro message is in a Kafka topic, you will need to write a Consumer which can retrieve the avro message from the topic. The nature of the message doesn’t really matter in terms of retrieving the message. The Spark Streaming documentation provides an example of how to setup a consumer to retrieve events from Kafka topics. You could expand the example to process the avro message.

https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html