Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

kafka-consumer-groups.sh vs ConsumerOffsetChecker

Highlighted

kafka-consumer-groups.sh vs ConsumerOffsetChecker

Expert Contributor

Hi All - I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying to use the ConsumerOffsetChecker & bin/kafka-consumer-groups.sh to check for offsets.

I'm seeing different behavior in using the above 2 scrripts.

Here is what i did -> a) When i use ConsumerOffsetChecker

$KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper localhost:2181 --group myGroup --topic newBroker1

[2017-06-15 11:50:23,729] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)

Group Topic Pid Offset logSize Lag Owner

myGroup newBroker1 0 224216 507732 283516 none

myGroup newBroker1 1 224888 508978 284090 none

myGroup newBroker1 2 141104 424038 282934 none

myGroup newBroker1 3 11829 295110 283281 none

myGroup newBroker1 4 11580 294510 282930 none

b) When i run the kafka-consumer-groups.sh (since the WARNING message above says - ConsumerOffsetChecker is deprecated & we need to use ConsumerGroupCommand instead)

$KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092,localhost:9093,localhost:9094,localhost:9095 --new-consumer --group myGroup

Consumer group `myGroup` does not exist or is rebalancing.

OR

$KAFKA_HOME/bin/kafka-consumer-groups.sh --describe --zookeeper localhost:2181 --group myGroup

No topic available for consumer group provided

GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER

The question - When i use $KAFKA_HOME/bin/kafka-consumer-groups.sh - why is the script not giving me Offset details of the group - myGroup & Topic - newBroker1

What needs to be done for this (using script $KAFKA_HOME/bin/kafka-consumer-groups.sh) ?

2 REPLIES 2

Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker

Expert Contributor

@mqureshi, @amankumbare - looping you in, any ideas ?

Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker

Super Guru
@Karan Alang

What you are doing is right. "--new-consumer" should make it work if you are using the new consumer. Can you please try again the following.

kafka-run-class.sh kafka.admin.ConsumerGroupCommand --new-consumer --describe --group myGroup --zookeeper localhost:2181

But if this doesn't help, check your zookeeper.

Run zookeeper cli. do an ls on /consumers/myGroup/offsets/<topicname>

You should get some thing like [0].

Then run a get command on this offset. Something like:

get /consumers/myGroup/offsets/<topicname>/0

Then you'll see the first thing is your offset. If it's there, then paste the result and we'll go from there.

Don't have an account?
Coming from Hortonworks? Activate your account here