Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Reading JSON data from Kafka in SAM (HDF 3.1)

Solved Go to solution

Reading JSON data from Kafka in SAM (HDF 3.1)

New Contributor

Hi All,I am exploring around Streaming Analytics Manager, I have a flow which read messages from Kafka topic and writes it back to another Kafka topic after filtering based on some rules.

above flow works when messages in Kafka are in Avro format but it fails in case of JSON with following error
"SAM error: com.hortonworks.registries.schemaregistry.serde.SerDesException: Unknown protocol id [123] received while deserializing the payload"

I have checked https://community.hortonworks.com/questions/121234/sam-error-comhortonworksregistriesschemaregistrys... which reported it as a bug in 3.0 later resolved in 3.1. I am using 3.1 Sandbox.

I have also seen Community Hello -World SAM application saying messages must be in Avro format. Is that true that SAM kafka Source will work only for messages in Avro format?

Please help me to resolve the issue

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Reading JSON data from Kafka in SAM (HDF 3.1)

New Contributor

If you check the documentation for the Kafka Source, you will see that the 'kafka topic' field needs to pull the schema from the Schema Registry.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_streaming-analytics-manager-user-guide/c...

Meanwhile, the Schema Registry documentation notes that AVRO is the only supported type of schema at this time.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_schema-registry-user-guide/content/creat...

So it's possible in the future more formats will be allowed, but not at the moment. Good luck!

2 REPLIES 2

Re: Reading JSON data from Kafka in SAM (HDF 3.1)

New Contributor

If you check the documentation for the Kafka Source, you will see that the 'kafka topic' field needs to pull the schema from the Schema Registry.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_streaming-analytics-manager-user-guide/c...

Meanwhile, the Schema Registry documentation notes that AVRO is the only supported type of schema at this time.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_schema-registry-user-guide/content/creat...

So it's possible in the future more formats will be allowed, but not at the moment. Good luck!

Re: Reading JSON data from Kafka in SAM (HDF 3.1)

New Contributor

Thank you for the clarification. @Nick Lewis