NiFi 1.2: How does Kafka Producer Connect to Schema Registry

[#1]

Hello,

I'd like to publish avro message to a kafka topic, with the schema registered in avro schema registry.

However, I didn't find any KafkaProducer processors have a "Schema Registry URL".

Please note my downstream is not in NiFi. It can be a independent java app implemented the Kafka consumer client api, which consumes the avro message and connects to Schema registry to retrieve the schema.

It seems in v1.2 the schema registry is integrated within NiFi, is it? How does the other apps share schemas with NiFi ?

Any documents or examples for using schema registry with Kafka processors in NiFi?

Thanks.

[#2 · Accepted solution]

In Apache NiFi 1.2, there are is PublishKafkaRecord_0_10 which uses a RecordReader service for reading incoming flow files as records, and a RecordWriter service for serializing records to bytes that will be published to a Kafka topic.

Each RecordReader and RecordWriter can be configured with a strategy for accessing a schema, one strategy is to get a schema by name from a schema registry. If using a schema registry, you can configure a schema registry service.

Apache NiFi provides AvroSchemaRegistry as a local schema registry that can be used within NiFi, or a HortonworksSchemaRegistry service which can be used with an external Hortonworks schema registry.

In addition, anyone could implement their own schema registry service to talk to any other schema registry or backing storage mechanism.

[#3]

Hi @Bryan Bende,

Thank you for your response.

I am more interested in PublishKafkaRecord_0_10 + an external schema registry.

Since we have already deployed ConfluentSchemaRegistry used by some other apps, we don't want to switch to HortonworksSchemaRegistry. We want NiFi workflow to share avro schema with the other apps.

In this case, do we have to implement a Confluent-based schema registry control service? or any easier solutions?

Thanks.

[#4]

Correct, you would have to implement a ConfluentSchemaRegistry controller service for NiFi.