RecordTooLargeException on large messages in Kafka?

Question

I am using HDP-2.6.5.0 with kafka 1.0.0; I have to process large (16M) messages so i set

message.max.bytes=18874368replica.fetch.max.bytes = 18874368socket.request.max.bytes =18874368

From Ambary/Kafka configs screen and restarted Kafka services

When I try to send 16M messages:

/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list <broker-ip>:6667 --topic test < ./big.txt

I still have the same error:

ERROR Error when sending message to topic test with key: null, value: 16777239 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.RecordTooLargeException: The message is 16777327 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

I tried to set max.request.size in producer.properties file but still have the same error. What am I missing?

Thank you,

jsensharma · Accepted Answer

@lvic4594_ As you are keep getting "RecordTooLargeException" even after increasing few properties that you listed in your previous comment. So can you please let us know exactly where are you noticing those exceptions? Broker side or Producer side or Consumer side?Also can you please try to specify the complete path of the "producer.properties" file in the "kafka-console-producer.sh" command line just to ensure that we are using the correct producer properties file?Example:/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list :6667 --producer.config /etc/kafka/conf/producer.properties --topic test < ./big.txt.Also please verify if this file has the correct value:# grep 'max.request.size' /etc/kafka/conf/producer.properties.Reference Article: https://community.cloudera.com/t5/Community-Articles/Kafka-producer-running-into-multiple-org-apache-kafka-common/ta-p/248636Broker side: "message.max.bytes" - this is the largest size of the message that can be received by the broker from a producer. "replica.fetch.max.bytes" - The number of bytes of messages to attempt to fetch for each partition.Producer side: "max.request.size" is a limit to send the larger message.Consumer side: Increase "max.partition.fetch.bytes" which help you to consume big messages. max number of bytes per partition returned by the server. should be larger than the max.message.size so consumer can read the largest message sent by the broker. For Consumer side can you please let us know if you have also increased the "max.partition.fetch.bytes" ?

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.