Kafka ConsumerGroupCommand Error

[#1]

Hi,

I am on HDP-2.5.5 and Kafka 0.10.0. Cluster is Kerberised. When I run the new ConsumerGroupCommand I get the error below:

ubuntu@hds-01:~$ /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --security-protocol PLAINTEXTSASL --new-consumer --describe --group spoutconsumer

[2017-09-17 19:43:54,039] WARN TGT refresh thread time adjusted from Sun Sep 17 12:49:30 UTC 2017 to Sun Sep 17 19:44:54 UTC 2017 since the former is sooner than the minimum refresh interval (60 seconds) from now. (org.apache.kafka.common.security.kerberos.KerberosLogin)
[2017-09-17 19:43:54,526] WARN Bootstrap broker hds-02.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-09-17 19:43:54,533] WARN Bootstrap broker hds-04.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-09-17 19:43:54,535] WARN Bootstrap broker hds-01.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-09-17 19:43:54,638] WARN Bootstrap broker hds-03.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
Error while executing consumer group command Request GROUP_COORDINATOR failed on brokers List(hds-02.field.hortonworks.com:6667 (id: -2 rack: null), hds-04.field.hortonworks.com:6667 (id: -4 rack: null), hds-01.field.hortonworks.com:6667 (id: -1 rack: null), hds-03.field.hortonworks.com:6667 (id: -3 rack: null))
java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(hds-02.field.hortonworks.com:6667 (id: -2 rack: null), hds-04.field.hortonworks.com:6667 (id: -4 rack: null), hds-01.field.hortonworks.com:6667 (id: -1 rack: null), hds-03.field.hortonworks.com:6667 (id: -3 rack: null))
	at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)
	at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72)
	at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125)
	at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147)
	at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:320)
	at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:97)
	at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:308)
	at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:76)
	at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
<br>

What is wrong here? The user has valid ticket.

I am quite sure the --security-protocol PLAINTEXTSASL should be OK, cause it works on both a console consumer and a producer.

[#2]

Hi Jasper,

I don't think you have a problem while logging in.

https://github.com/streamsets/datacollector/blob/master/apache-kafka_0_9-lib/src/main/java/org/apach...

After skimming through the above link, I guess if there was any error w.r.t. having a valid ticket, you would've got a log in KerberosLogin itself.

What principal are you using?

Can you check contents of kafka_client_jaas.conf. Is it of the form below:

KafkaClient

{ com.sun.security.auth.module.Krb5LoginModule required

useTicketCache=true; };

Or you have keytab configuration?

If former, please see kafka_jaas.conf for Client section and kinit with user/keytab mentioned there. Try running the command again as:

/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --security-protocol PLAINTEXTSASL --new-consumer --describe --group spoutconsumer -Djava.security.auth.login.config=/etc/kafka/kafka_jaas.conf

If above command does not work, try exporting that variable.

[#3]

@tsharma

Thanks for your suggestion. It turns out the kafka-consumer-groups.sh eventually uses :

KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_jaas.conf

The contents of this file is:

KafkaServer {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/etc/security/keytabs/kafka.service.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="kafka"
   principal="kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM";
};
KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useTicketCache=true
   renewTicket=true
   serviceName="kafka";
};
Client {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/etc/security/keytabs/kafka.service.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="zookeeper"
   principal="kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM";
};

So it does not make a difference what principal is used to kinit. Even when I take the kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM principal I get the same result:

Error while executing consumer group command Request GROUP_COORDINATOR failed on brokers List(dem-mtrn-hds-01.field.hortonworks.com:6667 (id: -1 rack: null), dem-mtrn-hds-04.field.hortonworks.com:6667 (id: -4 rack: null), dem-mtrn-hds-02.field.hortonworks.com:6667 (id: -2 rack: null), dem-mtrn-hds-03.field.hortonworks.com:6667 (id: -3 rack: null))
java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(dem-mtrn-hds-01.field.hortonworks.com:6667 (id: -1 rack: null), dem-mtrn-hds-04.field.hortonworks.com:6667 (id: -4 rack: null), dem-mtrn-hds-02.field.hortonworks.com:6667 (id: -2 rack: null), dem-mtrn-hds-03.field.hortonworks.com:6667 (id: -3 rack: null))
	at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)
	at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72)
	at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125)
	at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147)
	at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:320)
	at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:97)
	at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:308)
	at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:76)
	at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)

[#4 · Accepted solution]

It is solved now. Turns out that the ' --security-protocol PLAINTEXTSASL ' is not honored by the kafka-consumer-groups.sh tool.

But it does work if you provide the same through the --command-config flag like:

echo "security.protocol=PLAINTEXTSASL" > /tmp/consgroupcmd.prop
/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --new-consumer --describe --group console-consumer-spout1 --command-config /tmp/consgroupcmd.prop

But now, to my suprise it turns out that the kafka-consumer-groups.sh / kafka.admin.ConsumerGroupCommand tool does NOT recognize any consumer group of the --new-consumer type for which offsets are stored on Kafka.

Good old kafka-consumer-offset-checker.sh is still the go-to tool for this (works both on zookeeper and Kafka kept offsets):

/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-offset-checker.sh --zookeeper $ZOOKEEPER_HOSTS --security-protocol PLAINTEXTSASL --topic spout_test --group console-consumer-spout1
KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/2.5.5.0-157/kafka/config/kafka_client_jaas.conf
[2017-09-20 21:13:30,168] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
Group           Topic                          Pid Offset          logSize         Lag             Owner
console-consumer-spout1 spout_test                     0   36156           68201           32045           none
console-consumer-spout1 spout_test                     1   0               68198           68198           none
console-consumer-spout1 spout_test                     2   12053           68198           56145           none
console-consumer-spout1 spout_test                     3   0               68200           68200           none

It is weird that the deprecation warning recommends to use ConsumerGroupCommand while this does not even work on the new-consumer Kafka offsets:

/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --new-consumer --describe --group console-consumer-spout1 --command-config /tmp/consgroupcmd.prop
KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_jaas.conf
Consumer group `console-consumer-spout1` does not exist or is rebalancing.
[2017-09-20 21:11:04,189] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)

The group 'console-consumer-spout1' does exist as can be seen in the output of the previous command...

[#5]

Reason kafka-consumer-groups.sh returned "Consumergroup`console-consumer-spout1` does not exist or is rebalancing." was that the consumergroup was not active at that moment. If it is not consuming, it is not shown with this command.

In that respect kafka-consumer-offset-checker.sh does a better job as it always returns the offsets, consumer active or not