Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi Putkafka processor Maximum allowed data size exceeded

avatar
Expert Contributor

Hello,

I recently began receiving the following error in the NiFi PutKafka processor:

PutKafka[id=5ab0d936-4326-37cc-9932-3f17d28f8d7e] PutKafka[id=5ab0d936-4326-37cc-9932-3f17d28f8d7e] failed to process due to java.lang.IllegalStateException: Maximum allowed data size of 1048576 exceeded.; rolling back session: java.lang.IllegalStateException: Maximum allowed data size of 1048576 exceeded.

Is there any way to handle larger messages via the PutKafka processor in NiFi? I already set up the the PutKafka processor with Max Record Size to 10MB, but it is still giving the error. Any thoughts on how to fix this issue would be highly appreciated.

1 ACCEPTED SOLUTION

avatar
Master Guru

This is a known problem with PutKafka: https://issues.apache.org/jira/browse/NIFI-2671

What versions of Kafka and NiFi are you using?

PutKafka should only be used with a 0.8 Kafka broker, PublishKafka and PublishKafka_0_10 should be used with Kafka 0.9 and 0.10 respectively.

For PublishKafka and PublishKafka_0_10, they were fixed to send messages larger than 1MB:

https://issues.apache.org/jira/browse/NIFI-2614

View solution in original post

2 REPLIES 2

avatar
Master Guru

This is a known problem with PutKafka: https://issues.apache.org/jira/browse/NIFI-2671

What versions of Kafka and NiFi are you using?

PutKafka should only be used with a 0.8 Kafka broker, PublishKafka and PublishKafka_0_10 should be used with Kafka 0.9 and 0.10 respectively.

For PublishKafka and PublishKafka_0_10, they were fixed to send messages larger than 1MB:

https://issues.apache.org/jira/browse/NIFI-2614

avatar
Expert Contributor

Hello @Bryan Bende , I am using NiFi-1.0.0 and kafka-0.9. I will do the changes from the PutKafka processor to the PublishKafka processor. Thanks for informing me that the issue has been fixed for PublishKafka and PublishKafka_0_10.