NiFi Putkafka processor Maximum allowed data size exceeded

[#1]

Hello,

I recently began receiving the following error in the NiFi PutKafka processor:

PutKafka[id=5ab0d936-4326-37cc-9932-3f17d28f8d7e] PutKafka[id=5ab0d936-4326-37cc-9932-3f17d28f8d7e] failed to process due to java.lang.IllegalStateException: Maximum allowed data size of 1048576 exceeded.; rolling back session: java.lang.IllegalStateException: Maximum allowed data size of 1048576 exceeded.

Is there any way to handle larger messages via the PutKafka processor in NiFi? I already set up the the PutKafka processor with Max Record Size to 10MB, but it is still giving the error. Any thoughts on how to fix this issue would be highly appreciated.

[#2 · Accepted solution]

This is a known problem with PutKafka: https://issues.apache.org/jira/browse/NIFI-2671

What versions of Kafka and NiFi are you using?

PutKafka should only be used with a 0.8 Kafka broker, PublishKafka and PublishKafka_0_10 should be used with Kafka 0.9 and 0.10 respectively.

For PublishKafka and PublishKafka_0_10, they were fixed to send messages larger than 1MB:

https://issues.apache.org/jira/browse/NIFI-2614

[#3]

Hello @Bryan Bende , I am using NiFi-1.0.0 and kafka-0.9. I will do the changes from the PutKafka processor to the PublishKafka processor. Thanks for informing me that the issue has been fixed for PublishKafka and PublishKafka_0_10.