Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

SAM error: com.hortonworks.registries.schemaregistry.serde.SerDesException: Unknown protocol id [123] received while deserializing the payload at com.hortonworks.registries.schemaregistry.serdes.avro.AvroSnapsh

avatar
Guru

Simple SAM flow: Kafka -> (Storm) filter -> Kafka

Fails at Storm, which reports: com.hortonworks.registries.schemaregistry.serde.SerDesException: Unknown protocol id [123] received while deserializing the payload at com.hortonworks.registries.schemaregistry.serdes.avro.AvroSnapsh

Wondering what could cause this. (Schema seems properly configured)

1 ACCEPTED SOLUTION

avatar

@Timothy Spann @Greg Keys

This is a known issue with HDF-3.0.1 release and we fixed did a release HDF-3.0.1.1. Please find the details here

http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_release-notes/content/c....

1. Take a backup of existing SAM app. You can export the app to your local computer.

2. Upgrade the cluster to 3.0.1.1

3. If you have existing data in a Kafka topic that was ingested with the above bug. Make sure you create a new topic or delete the existing topic.

4. Import the SAM topology, make sure you configure the topic and other cluster details in the app.

View solution in original post

8 REPLIES 8

avatar
Expert Contributor

avatar
Master Guru

1. Make sure NiFi and SAM are the same version HDF

2. Make sure kafka message is converted properly to Avro.

3. Restart Kafka, SAM, NiFi

4. Make sure all the settings are proper.

avatar
Master Guru

Make sure your queue is empty, old bad messages might be sent.

I would create a new empty queue add one checked valid message (check your avro message with avro tools) and send that.. No nulls!

avatar

@Timothy Spann @Greg Keys

This is a known issue with HDF-3.0.1 release and we fixed did a release HDF-3.0.1.1. Please find the details here

http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_release-notes/content/c....

1. Take a backup of existing SAM app. You can export the app to your local computer.

2. Upgrade the cluster to 3.0.1.1

3. If you have existing data in a Kafka topic that was ingested with the above bug. Make sure you create a new topic or delete the existing topic.

4. Import the SAM topology, make sure you configure the topic and other cluster details in the app.

avatar
Guru

Followed instructions and it worked. Thanks @Sriharsha Chintalapani

avatar
Contributor

@Greg Keys I am having the same issue with hdf 3.3.1. I have checked the schema file and the input file as well.
I have done what was mentioned by @Sriharsha Chintalapani
Schema

{
    "namespace": "hdf.heaptrace.com",
    "type": "record",
    "name": "PatientField",
    "fields": [
    {"name": "Patient_name","type": "string"}
  ]
}

JSON data

{"Patient_name":"john"}

Please help !!!

I have converted data from json to avro and then back again as well using avro tools.

avatar
Explorer

@Sriharsha Chintalapani I getting the same error, cluster update didn't solve the issue.

I'm trying to consume plain CSV(roshan,22) to Streamline via NIFI(to convert CSV to AVRO). Kafka > NIFI > Kafka. From Storm I'm getting the error similar to above.

com.hortonworks.registries.schemaregistry.serde.SerDesException: Unknown protocol id [114] received while deserializing the payload at com.hortonworks.registries.schemaregistry.serdes.avro.AvroSnapsh

My Flow is as follows, One branch directly to a Kafka topic and other one to serialize and publish to kafka. After running data through the PublishKafkaRecord the output simply removes the "," (roshan,22 turns to roshan22) and the above mentioned error appears in Storm. I'm really new to this stack any help would be appreciated.

40453-screen-shot-2017-09-21-at-123647-pm.png

Avro schema

{
 "type": "record",
 "name": "user",
 "fields": [
  {
   "name": "name",
   "type": "string",
   "default": null
  },
  {
   "name": "age",
   "type": "string",
   "default": null
  }
 ]
}

Update Attribute Processor

40454-screen-shot-2017-09-21-at-124058-pm.png

Publish Kafka Record Processor

40455-screen-shot-2017-09-21-at-124147-pm.png

CSVReader Controller Service

40456-screen-shot-2017-09-21-at-124413-pm.png

AvroRecordSetWriter Controller Service

40457-screen-shot-2017-09-21-at-124459-pm.png

avatar
Contributor

Have you resolved your issue ?? How did you do it ??