Kafka ConsumerGroupCommand Error

[#1]

Hi,

I am on HDP-2.5.5 and Kafka 0.10.0. Cluster is Kerberised. When I run the new ConsumerGroupCommand I get the error below:

ubuntu@hds-01:~$ /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --security-protocol PLAINTEXTSASL --new-consumer --describe --group spoutconsumer

[2017-09-17 19:43:54,039] WARN TGT refresh thread time adjusted from Sun Sep 17 12:49:30 UTC 2017 to Sun Sep 17 19:44:54 UTC 2017 since the former is sooner than the minimum refresh interval (60 seconds) from now. (org.apache.kafka.common.security.kerberos.KerberosLogin)
[2017-09-17 19:43:54,526] WARN Bootstrap broker hds-02.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-09-17 19:43:54,533] WARN Bootstrap broker hds-04.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-09-17 19:43:54,535] WARN Bootstrap broker hds-01.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
[2017-09-17 19:43:54,638] WARN Bootstrap broker hds-03.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient)
Error while executing consumer group command Request GROUP_COORDINATOR failed on brokers List(hds-02.field.hortonworks.com:6667 (id: -2 rack: null), hds-04.field.hortonworks.com:6667 (id: -4 rack: null), hds-01.field.hortonworks.com:6667 (id: -1 rack: null), hds-03.field.hortonworks.com:6667 (id: -3 rack: null))
java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(hds-02.field.hortonworks.com:6667 (id: -2 rack: null), hds-04.field.hortonworks.com:6667 (id: -4 rack: null), hds-01.field.hortonworks.com:6667 (id: -1 rack: null), hds-03.field.hortonworks.com:6667 (id: -3 rack: null))
	at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)
	at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72)
	at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125)
	at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147)
	at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:320)
	at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:97)
	at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:308)
	at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:76)
	at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
<br>

What is wrong here? The user has valid ticket.

I am quite sure the --security-protocol PLAINTEXTSASL should be OK, cause it works on both a console consumer and a producer.

[#2]

Hi Jasper,

I don't think you have a problem while logging in.

https://github.com/streamsets/datacollector/blob/master/apache-kafka_0_9-lib/src/main/java/org/apach...

After skimming through the above link, I guess if there was any error w.r.t. having a valid ticket, you would've got a log in KerberosLogin itself.

What principal are you using?

Can you check contents of kafka_client_jaas.conf. Is it of the form below:

KafkaClient

{ com.sun.security.auth.module.Krb5LoginModule required

useTicketCache=true; };

Or you have keytab configuration?

If former, please see kafka_jaas.conf for Client section and kinit with user/keytab mentioned there. Try running the command again as:

/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --security-protocol PLAINTEXTSASL --new-consumer --describe --group spoutconsumer -Djava.security.auth.login.config=/etc/kafka/kafka_jaas.conf

If above command does not work, try exporting that variable.

[#3]

@tsharma

Thanks for your suggestion. It turns out the kafka-consumer-groups.sh eventually uses :

KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_jaas.conf

The contents of this file is:

KafkaServer {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/etc/security/keytabs/kafka.service.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="kafka"
   principal="kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM";
};
KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useTicketCache=true
   renewTicket=true
   serviceName="kafka";
};
Client {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="/etc/security/keytabs/kafka.service.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="zookeeper"
   principal="kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM";
};

So it does not make a difference what principal is used to kinit. Even when I take the kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM principal I get the same result:

Error while executing consumer group command Request GROUP_COORDINATOR failed on brokers List(dem-mtrn-hds-01.field.hortonworks.com:6667 (id: -1 rack: null), dem-mtrn-hds-04.field.hortonworks.com:6667 (id: -4 rack: null), dem-mtrn-hds-02.field.hortonworks.com:6667 (id: -2 rack: null), dem-mtrn-hds-03.field.hortonworks.com:6667 (id: -3 rack: null))
java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(dem-mtrn-hds-01.field.hortonworks.com:6667 (id: -1 rack: null), dem-mtrn-hds-04.field.hortonworks.com:6667 (id: -4 rack: null), dem-mtrn-hds-02.field.hortonworks.com:6667 (id: -2 rack: null), dem-mtrn-hds-03.field.hortonworks.com:6667 (id: -3 rack: null))
	at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)
	at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72)
	at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125)
	at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147)
	at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:320)
	at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:97)
	at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:308)
	at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:76)
	at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)

[#4]

Reason kafka-consumer-groups.sh returned "Consumergroup`console-consumer-spout1` does not exist or is rebalancing." was that the consumergroup was not active at that moment. If it is not consuming, it is not shown with this command.

In that respect kafka-consumer-offset-checker.sh does a better job as it always returns the offsets, consumer active or not