question Re: RecordTooLargeException on large messages in Kafka? in Support Questions

RecordTooLargeException on large messages in Kafka?

lvic4594_ — Tue, 13 Aug 2019 22:37:41 GMT

I am using HDP-2.6.5.0 with kafka 1.0.0; I have to process large (16M) messages so i set

message.max.bytes=18874368replica.fetch.max.bytes = 18874368socket.request.max.bytes =18874368

From Ambary/Kafka configs screen and restarted Kafka services

When I try to send 16M messages:

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list <broker-ip>:6667 --topic test < ./big.txt

I still have the same error:

ERROR Error when sending message to topic test with key: null, value: 16777239 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

org.apache.kafka.common.errors.RecordTooLargeException: The message is 16777327 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

I tried to set max.request.size in producer.properties file but still have the same error. What am I missing?

Thank you,

Re: RecordTooLargeException on large messages in Kafka?

jsensharma — Mon, 19 Aug 2019 08:28:51 GMT

@lvic4594_

As you are keep getting "RecordTooLargeException" even after increasing few properties that you listed in your previous comment.

So can you please let us know exactly where are you noticing those exceptions? Broker side or Producer side or Consumer side?

Also can you please try to specify the complete path of the "producer.properties" file in the "kafka-console-producer.sh" command line just to ensure that we are using the correct producer properties file?

Example:

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list <broker-ip>:6667 --producer.config /etc/kafka/conf/producer.properties --topic test < ./big.txt

.
Also please verify if this file has the correct value:

# grep 'max.request.size' /etc/kafka/conf/producer.properties

Reference Article: https://community.cloudera.com/t5/Community-Articles/Kafka-producer-running-into-multiple-org-apache-kafka-common/ta-p/248636

Broker side: "message.max.bytes" - this is the largest size of the message that can be received by the broker from a producer.
"replica.fetch.max.bytes" - The number of bytes of messages to attempt to fetch for each partition.

Producer side: "max.request.size" is a limit to send the larger message.

Consumer side: Increase "max.partition.fetch.bytes" which help you to consume big messages. max number of bytes per partition returned by the server. should be larger than the max.message.size so consumer can read the largest message sent by the broker.

For Consumer side can you please let us know if you have also increased the "max.partition.fetch.bytes" ?

Re: RecordTooLargeException on large messages in Kafka?

lvic4594_ — Mon, 19 Aug 2019 18:58:29 GMT

--producer-config did the trick for kafka-console-producer.sh...Or changing "max.request.size" directly in producer code. I didn't have to modify consumer settings

Re: RecordTooLargeException on large messages in Kafka?

jsensharma — Tue, 20 Aug 2019 00:26:49 GMT

@lvic4594_ Great to know that the issue is resolved after making the recommended changes to use the producer.config argument explicitly

--producer.config /etc/kafka/conf/producer.properties

As the issue is resolved, hence it will be great to mark this thread as Solved. So that other users can quickly find the resolved threads/answers,.

Re: RecordTooLargeException on large messages in Kafka?

lvic4594_ — Wed, 21 Aug 2019 13:30:17 GMT

it will be great to mark this thread as Solved

Would be happy to, but don't see this available in "options" - i can only see "mark as read"

Re: RecordTooLargeException on large messages in Kafka?

cjervis — Thu, 22 Aug 2019 15:14:01 GMT

@lvic4594_ Marking the solution is easy. The author of the original question will see an Accept as solution button on every reply. Click the button on the reply(s) that solved the issue to mark it as such.

Re: RecordTooLargeException on large messages in Kafka?

lvic4594_ — Thu, 22 Aug 2019 18:07:01 GMT

Thanks, i didn't realized i am not actually logged at the moment