Reading JSON data from Kafka in SAM (HDF 3.1)

[#1]

Hi All,I am exploring around Streaming Analytics Manager, I have a flow which read messages from Kafka topic and writes it back to another Kafka topic after filtering based on some rules.

above flow works when messages in Kafka are in Avro format but it fails in case of JSON with following error
"SAM error: com.hortonworks.registries.schemaregistry.serde.SerDesException: Unknown protocol id [123] received while deserializing the payload"

I have checked https://community.hortonworks.com/questions/121234/sam-error-comhortonworksregistriesschemaregistrys... which reported it as a bug in 3.0 later resolved in 3.1. I am using 3.1 Sandbox.

I have also seen Community Hello -World SAM application saying messages must be in Avro format. Is that true that SAM kafka Source will work only for messages in Avro format?

Please help me to resolve the issue

[#2 · Accepted solution]

If you check the documentation for the Kafka Source, you will see that the 'kafka topic' field needs to pull the schema from the Schema Registry.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_streaming-analytics-manager-user-guide/c...

Meanwhile, the Schema Registry documentation notes that AVRO is the only supported type of schema at this time.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_schema-registry-user-guide/content/creat...

So it's possible in the future more formats will be allowed, but not at the moment. Good luck!

[#3]

Thank you for the clarification. @Nick Lewis