question Reading JSON data from Kafka in SAM (HDF 3.1) in Archives of Support Questions (Read Only)

Reading JSON data from Kafka in SAM (HDF 3.1)

shivkanaujiya00 — Wed, 18 Jul 2018 15:10:23 GMT

Hi All,I am exploring around Streaming Analytics Manager, I have a flow which read messages from Kafka topic and writes it back to another Kafka topic after filtering based on some rules.

above flow works when messages in Kafka are in Avro format but it fails in case of JSON with following error
"SAM error: com.hortonworks.registries.schemaregistry.serde.SerDesException: Unknown protocol id [123] received while deserializing the payload"

I have checked https://community.hortonworks.com/questions/121234/sam-error-comhortonworksregistriesschemaregistryse.html which reported it as a bug in 3.0 later resolved in 3.1. I am using 3.1 Sandbox.

I have also seen Community Hello -World SAM application saying messages must be in Avro format. Is that true that SAM kafka Source will work only for messages in Avro format?

Please help me to resolve the issue

Re: Reading JSON data from Kafka in SAM (HDF 3.1)

nlewis135031 — Wed, 01 Aug 2018 03:07:41 GMT

If you check the documentation for the Kafka Source, you will see that the 'kafka topic' field needs to pull the schema from the Schema Registry.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_streaming-analytics-manager-user-guide/content/source-configurations.html

Meanwhile, the Schema Registry documentation notes that AVRO is the only supported type of schema at this time.

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_schema-registry-user-guide/content/create-schema-metadata.html

So it's possible in the future more formats will be allowed, but not at the moment. Good luck!

Re: Reading JSON data from Kafka in SAM (HDF 3.1)

shivkanaujiya00 — Tue, 07 Aug 2018 09:32:31 GMT

Thank you for the clarification. @Nick Lewis