- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
NiFi Putkafka processor Maximum allowed data size exceeded
- Labels:
-
Apache NiFi
Created 12-07-2016 04:34 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
I recently began receiving the following error in the NiFi PutKafka processor:
PutKafka[id=5ab0d936-4326-37cc-9932-3f17d28f8d7e] PutKafka[id=5ab0d936-4326-37cc-9932-3f17d28f8d7e] failed to process due to java.lang.IllegalStateException: Maximum allowed data size of 1048576 exceeded.; rolling back session: java.lang.IllegalStateException: Maximum allowed data size of 1048576 exceeded.
Is there any way to handle larger messages via the PutKafka processor in NiFi? I already set up the the PutKafka processor with Max Record Size to 10MB, but it is still giving the error. Any thoughts on how to fix this issue would be highly appreciated.
Created 12-07-2016 06:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is a known problem with PutKafka: https://issues.apache.org/jira/browse/NIFI-2671
What versions of Kafka and NiFi are you using?
PutKafka should only be used with a 0.8 Kafka broker, PublishKafka and PublishKafka_0_10 should be used with Kafka 0.9 and 0.10 respectively.
For PublishKafka and PublishKafka_0_10, they were fixed to send messages larger than 1MB:
Created 12-07-2016 06:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is a known problem with PutKafka: https://issues.apache.org/jira/browse/NIFI-2671
What versions of Kafka and NiFi are you using?
PutKafka should only be used with a 0.8 Kafka broker, PublishKafka and PublishKafka_0_10 should be used with Kafka 0.9 and 0.10 respectively.
For PublishKafka and PublishKafka_0_10, they were fixed to send messages larger than 1MB:
Created 12-07-2016 06:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @Bryan Bende , I am using NiFi-1.0.0 and kafka-0.9. I will do the changes from the PutKafka processor to the PublishKafka processor. Thanks for informing me that the issue has been fixed for PublishKafka and PublishKafka_0_10.
