Nifi: Kafka Producer with Avro format in both key and value

[#1]

Hello all,

We have a couple of flows already writting values into Kafka using Avro format for the value, via an AvroRecordSetWriter that also uses a Confluent Schema Registry.

The thing is that now we would also need to write the Kafka key also in Avro format.

Is there any easy way to do it, or how would we isolate the key, to use an Avro writer to write it in Avro and then merge it will the original flow file as an attribute to be used as key by the PublishKafka processor?

Thank you.

Best regards,

Miguel

[#2 · Accepted solution]

I haven't seen anyone do a key in avro, generally you want a simple key.   Why avro as a key?  Is this common for some use cases?   

[#3]

Hello @TimothySpann,

Actually at the time it was a requirement from our client which in the meantime we managed to convince to drop.

So it has been "fixed" by itself.

Thank you anyway for the contribution.

Best regards,

Miguel

[#4]

If it does come up again, post here and we can put in a JIRA.

[#5]

Hello. I encountered the same problem, but I need to pass avro as a key. Is it possible to do this in publishKafka? Now I’m generating an avro schema and using ExtractText I save it into an attribute, then I pass this attribute to the Kafka key, but it doesn’t work. The avro circuit breaks down. And it is not parsed by the agent. Help please.