Support Questions

Find answers, ask questions, and share your expertise

Nifi: Kafka Producer with Avro format in both key and value

avatar
Explorer

Hello all,

 

We have a couple of flows already writting values into Kafka using Avro format for the value, via an AvroRecordSetWriter that also uses a Confluent Schema Registry.

 

The thing is that now we would also need to write the Kafka key also in Avro format.

 

Is there any easy way to do it, or how would we isolate the key, to use an Avro writer to write it in Avro and then merge it will the original flow file as an attribute to be used as key by the PublishKafka processor?

 

Thank you.

Best regards,

Miguel

1 ACCEPTED SOLUTION

avatar
Master Guru

I haven't seen anyone do a key in avro, generally you want a simple key.   Why avro as a key?  Is this common for some use cases?   

View solution in original post

4 REPLIES 4

avatar
Master Guru

I haven't seen anyone do a key in avro, generally you want a simple key.   Why avro as a key?  Is this common for some use cases?   

avatar
Explorer

Hello @TimothySpann,

 

Actually at the time it was a requirement from our client which in the meantime we managed to convince to drop.

So it has been "fixed" by itself.

Thank you anyway for the contribution.

 

Best regards,

Miguel

avatar
Master Guru

If it does come up again, post here and we can put in a JIRA.

avatar
New Contributor

Hello. I encountered the same problem, but I need to pass avro as a key. Is it possible to do this in publishKafka? Now I’m generating an avro schema and using ExtractText I save it into an attribute, then I pass this attribute to the Kafka key, but it doesn’t work. The avro circuit breaks down. And it is not parsed by the agent. Help please.