Can I use same consumer group Id for two different topic? Also, how many consumer will be created in one group id?

[#1]

There are two different topic and each topic have 30 partitions and two different producer is producing data in these topics. Also, there are two different kafka consumer but with same group Id. Will it create any problem? Also, how many consumer instance will be created under one group id? Please help me to understand this.

[#2]

Hi Shailendra,

Having 2 Kafka consumers with the same group ID will be just fine. This is by design actually. Kafka internals will try to load balance the topic consumption between any consumers registering on the group ID.

The number of consumers per group ID is not bound to anything, you can have as many as you want. Usually the consuming application (like Storm) sets/decides this. For optimal performance you would typically have as many consumers registering for a group ID as there are topic partitions. That way, a consumer would not have to switch between multiple assigned partitions, but would just stick to consuming from the same partition during its life span, which is most efficient.

[#3]

@Shailendra Lohia

And yes, you could use the same group ID for many different topics, if you want. For Kafka that would be transparent, as it would just be totally different 'keys' (topic:consumer_group_id) to Kafka

[#4]

I have configured one consumer group_id associated with multiple (around 10)  topics listening and producing messages to each other - sometimes not listening messages  and producer saying produced successfully

Note : It's failing listening randomly like sometimes service1 won't listen sometimes service1 listen and service2 won't listen  and so on

[#5]

@Shailendra Lohia Please mark the question as answered if sufficiently answered, thank you