question Re: Modify properties for Get/Put Kafka in HDF? in Archives of Support Questions (Read Only)

Modify properties for Get/Put Kafka in HDF?

eporeil — Wed, 10 Feb 2016 02:28:12 GMT

I'm seeing "kafka.common.MessageSizeTooLargeException" in my GetKafka processor. Based on some searching, I've found that by changing my properties files to include "fetch.message.max.bytes=10485760", I can alleviate the issue when using the Kafka console consumer. I'm wondering if there is any where I can add this property in HDF to allow for large sized messages to be consumer.

Re: Modify properties for Get/Put Kafka in HDF?

nsabharwal — Wed, 10 Feb 2016 03:02:55 GMT

@jwitt

Re: Modify properties for Get/Put Kafka in HDF?

Shelton — Wed, 10 Feb 2016 03:51:23 GMT

@Eric OReilly You shouldn't put fetch.message.max.bytes to config/server.properties but to your ConsumerConfi see this doc If you are using console consumer you may pass a --consumer.config consumer.properties flag where consumer.properties file will contain this config value.

Re: Modify properties for Get/Put Kafka in HDF?

JoeWitt — Fri, 12 Feb 2016 10:03:18 GMT

In the Get/PutKafka processors you should be able to add a dynamic property called 'fetch.message.max.bytes' and set the value you need. The processor should allow you to add dynamic properties which map to Kafka consumer properties and it will pass them to the consumer/producer config as needed.