Nifi: Kafka Producer with Avro format in both key and value

Question

Hello all,

We have a couple of flows already writting values into Kafka using Avro format for the value, via an AvroRecordSetWriter that also uses a Confluent Schema Registry.

The thing is that now we would also need to write the Kafka key also in Avro format.

Is there any easy way to do it, or how would we isolate the key, to use an Avro writer to write it in Avro and then merge it will the original flow file as an attribute to be used as key by the PublishKafka processor?

Thank you.

Best regards,

Miguel

TimothySpann · Accepted Answer

I haven't seen anyone do a key in avro, generally you want a simple key.   Why avro as a key?  Is this common for some use cases?

AnikaBaykus · Answer

Hello. I encountered the same problem, but I need to pass avro as a key. Is it possible to do this in publishKafka? Now I’m generating an avro schema and using ExtractText I save it into an attribute, then I pass this attribute to the Kafka key, but it doesn’t work. The avro circuit breaks down. And it is not parsed by the agent. Help please.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.