Skip to main content
migu
migu
Silver ⭐️
February 17, 2023
Solved

Nifi: Kafka Producer with Avro format in both key and value

Hello all,

 

We have a couple of flows already writting values into Kafka using Avro format for the value, via an AvroRecordSetWriter that also uses a Confluent Schema Registry.

 

The thing is that now we would also need to write the Kafka key also in Avro format.

 

Is there any easy way to do it, or how would we isolate the key, to use an Avro writer to write it in Avro and then merge it will the original flow file as an attribute to be used as key by the PublishKafka processor?

 

Thank you.

Best regards,

Miguel

TimothySpann
TimothySpannAnswer
Platinum ⭐️⭐️⭐️
March 29, 2023

I haven't seen anyone do a key in avro, generally you want a simple key.   Why avro as a key?  Is this common for some use cases?   

View in thread

4 replies

TimothySpann
Platinum ⭐️⭐️⭐️
TimothySpannAnswer
Platinum ⭐️⭐️⭐️
March 29, 2023

I haven't seen anyone do a key in avro, generally you want a simple key.   Why avro as a key?  Is this common for some use cases?   

Translate
migu
miguAuthor
Silver ⭐️
March 30, 2023

Hello @TimothySpann,

 

Actually at the time it was a requirement from our client which in the meantime we managed to convince to drop.

So it has been "fixed" by itself.

Thank you anyway for the contribution.

 

Best regards,

Miguel

Translate
TimothySpann
Platinum ⭐️⭐️⭐️
TimothySpann
Platinum ⭐️⭐️⭐️
March 30, 2023

If it does come up again, post here and we can put in a JIRA.

Translate
AnikaBaykus
Silver ⭐️⭐️
October 17, 2024

Hello. I encountered the same problem, but I need to pass avro as a key. Is it possible to do this in publishKafka? Now I’m generating an avro schema and using ExtractText I save it into an attribute, then I pass this attribute to the Kafka key, but it doesn’t work. The avro circuit breaks down. And it is not parsed by the agent. Help please.

Translate
    Terms & ConditionsAccessibility statement