question Re: NiFi 1.2: How does Kafka Producer Connect to Schema Registry in Archives of Support Questions (Read Only)

NiFi 1.2: How does Kafka Producer Connect to Schema Registry

alvinuw — Thu, 08 Jun 2017 22:28:37 GMT

Hello,

I'd like to publish avro message to a kafka topic, with the schema registered in avro schema registry.

However, I didn't find any KafkaProducer processors have a "Schema Registry URL".

Please note my downstream is not in NiFi. It can be a independent java app implemented the Kafka consumer client api, which consumes the avro message and connects to Schema registry to retrieve the schema.

It seems in v1.2 the schema registry is integrated within NiFi, is it? How does the other apps share schemas with NiFi ?

Any documents or examples for using schema registry with Kafka processors in NiFi?

Thanks.

Re: NiFi 1.2: How does Kafka Producer Connect to Schema Registry

bbende — Fri, 09 Jun 2017 00:38:52 GMT

In Apache NiFi 1.2, there are is PublishKafkaRecord_0_10 which uses a RecordReader service for reading incoming flow files as records, and a RecordWriter service for serializing records to bytes that will be published to a Kafka topic.

Each RecordReader and RecordWriter can be configured with a strategy for accessing a schema, one strategy is to get a schema by name from a schema registry. If using a schema registry, you can configure a schema registry service.

Apache NiFi provides AvroSchemaRegistry as a local schema registry that can be used within NiFi, or a HortonworksSchemaRegistry service which can be used with an external Hortonworks schema registry.

In addition, anyone could implement their own schema registry service to talk to any other schema registry or backing storage mechanism.

Re: NiFi 1.2: How does Kafka Producer Connect to Schema Registry

alvinuw — Tue, 13 Jun 2017 01:26:31 GMT

Hi @Bryan Bende,

Thank you for your response.

I am more interested in PublishKafkaRecord_0_10 + an external schema registry.

Since we have already deployed ConfluentSchemaRegistry used by some other apps, we don't want to switch to HortonworksSchemaRegistry. We want NiFi workflow to share avro schema with the other apps.

In this case, do we have to implement a Confluent-based schema registry control service? or any easier solutions?

Thanks.

Re: NiFi 1.2: How does Kafka Producer Connect to Schema Registry

bbende — Tue, 13 Jun 2017 01:42:49 GMT

Correct, you would have to implement a ConfluentSchemaRegistry controller service for NiFi.