Support Questions
Find answers, ask questions, and share your expertise

NiFi Putkafka processor Maximum allowed data size exceeded

Rising Star

Hello,

I recently began receiving the following error in the NiFi PutKafka processor:

PutKafka[id=5ab0d936-4326-37cc-9932-3f17d28f8d7e] PutKafka[id=5ab0d936-4326-37cc-9932-3f17d28f8d7e] failed to process due to java.lang.IllegalStateException: Maximum allowed data size of 1048576 exceeded.; rolling back session: java.lang.IllegalStateException: Maximum allowed data size of 1048576 exceeded.

Is there any way to handle larger messages via the PutKafka processor in NiFi? I already set up the the PutKafka processor with Max Record Size to 10MB, but it is still giving the error. Any thoughts on how to fix this issue would be highly appreciated.

1 ACCEPTED SOLUTION

This is a known problem with PutKafka: https://issues.apache.org/jira/browse/NIFI-2671

What versions of Kafka and NiFi are you using?

PutKafka should only be used with a 0.8 Kafka broker, PublishKafka and PublishKafka_0_10 should be used with Kafka 0.9 and 0.10 respectively.

For PublishKafka and PublishKafka_0_10, they were fixed to send messages larger than 1MB:

https://issues.apache.org/jira/browse/NIFI-2614

View solution in original post

2 REPLIES 2

This is a known problem with PutKafka: https://issues.apache.org/jira/browse/NIFI-2671

What versions of Kafka and NiFi are you using?

PutKafka should only be used with a 0.8 Kafka broker, PublishKafka and PublishKafka_0_10 should be used with Kafka 0.9 and 0.10 respectively.

For PublishKafka and PublishKafka_0_10, they were fixed to send messages larger than 1MB:

https://issues.apache.org/jira/browse/NIFI-2614

Rising Star

Hello @Bryan Bende , I am using NiFi-1.0.0 and kafka-0.9. I will do the changes from the PutKafka processor to the PublishKafka processor. Thanks for informing me that the issue has been fixed for PublishKafka and PublishKafka_0_10.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.