Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How many consumer instances in a Kafka consumer group deployed in a single server

How many consumer instances in a Kafka consumer group deployed in a single server

Contributor

Hello guys,

If I only have a single consumer group deployed in a single server.

How many consumer instances should I assign to this consumer group?

Thanks.

6 REPLIES 6

Re: How many consumer instances in a Kafka consumer group deployed in a single server

Contributor

@Alvin Jin - what is your partition size. you can start with consumer instance based on your partition startagey.

Re: How many consumer instances in a Kafka consumer group deployed in a single server

Contributor

Hi @njayakumar

Let's say I have 10 partitions for a topic.

But I have below config for my hardware in this server.

Thread(s) per core: 1

Core(s) per socket: 2

Socket(s): 1

Should I set 10(based on partitions) or 2(based on threads) consumer instances?

Thanks.

Highlighted

Re: How many consumer instances in a Kafka consumer group deployed in a single server

Contributor

@Alvin Jin - what is your kafka cluster size ? could you explain - how did you arrive the partition number ?

set num.io.threads to at least no.of disks

Please find below document for your reference

http://kafka.apache.org/documentation.html#diskandfs

Re: How many consumer instances in a Kafka consumer group deployed in a single server

Contributor

Hi @njayakumar

The Kafka cluster is 5 nodes.

However, we only deploy our consumer app in a single server rather than in a distributed way.

So I was wondering if this consumer app is a customer group, how many consumer should we set for it? based on threads of the machine?

Thanks.

Re: How many consumer instances in a Kafka consumer group deployed in a single server

Contributor

@Alvin Jin - You are data is distributed, producer is distributed but consumer is restricted to one box. am i missing anything ?

Re: How many consumer instances in a Kafka consumer group deployed in a single server

Contributor

Hi @njayakumar,

You are correct. Since we modify the legacy app to consume from kafka cluster,

the consumer is in one box.

Thanks.