Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How do i read kafka avro messages in spark streaming?

How do i read kafka avro messages in spark streaming?

New Contributor

Hi,

Does anyone know how to read the avro messages from kafka in spark streaming? Kindly comment or PM me.

Regards, Sathwik

1 REPLY 1
Highlighted

Re: How do i read kafka avro messages in spark streaming?

Expert Contributor

Your question leads me believe that you know how to write avro messages into a Kafka topic. Once your avro message is in a Kafka topic, you will need to write a Consumer which can retrieve the avro message from the topic. The nature of the message doesn’t really matter in terms of retrieving the message. The Spark Streaming documentation provides an example of how to setup a consumer to retrieve events from Kafka topics. You could expand the example to process the avro message.

https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html