ConsumeKafka2.0 not consuming right amount of records

[#1]

Hi,

I'm having issues with ConsumeKafka2.0 where i give it a Max Poll Records config of 10k but it never ends up consuming more than ~1k records, even if I use a long Max Uncommitted Time (e.g. 60s).

Is there any other config I should change to enable this?

[#2]

From Kafka perspective max.poll.records is an upper bound property in the number of messages that can be retrieved in a single poll call, a consumer is constantly consuming messages, for example, imagine that you have a topic and you send 10 messages, if max.poll.records are 10000 then the messages would never be consumed, so for the same reason, this is just an upper bound that's usually configured when the consumers start timing out because the processing of those messages is not happening in the max.poll.interval.ms (default 5 minutes).

In summary, consumers are constantly consuming messages (1 or many), and max.poll.records is just an upper bound property used to control the number of messages we can get in each poll call to make sure these messages are processed on time (max.pol.interval.ms).

Hope that information clarifies the usage of that property.