Created 09-18-2017 07:35 PM
Hi,
I am on HDP-2.5.5 and Kafka 0.10.0. Cluster is Kerberised. When I run the new ConsumerGroupCommand I get the error below:
ubuntu@hds-01:~$ /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --security-protocol PLAINTEXTSASL --new-consumer --describe --group spoutconsumer [2017-09-17 19:43:54,039] WARN TGT refresh thread time adjusted from Sun Sep 17 12:49:30 UTC 2017 to Sun Sep 17 19:44:54 UTC 2017 since the former is sooner than the minimum refresh interval (60 seconds) from now. (org.apache.kafka.common.security.kerberos.KerberosLogin) [2017-09-17 19:43:54,526] WARN Bootstrap broker hds-02.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient) [2017-09-17 19:43:54,533] WARN Bootstrap broker hds-04.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient) [2017-09-17 19:43:54,535] WARN Bootstrap broker hds-01.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient) [2017-09-17 19:43:54,638] WARN Bootstrap broker hds-03.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient) Error while executing consumer group command Request GROUP_COORDINATOR failed on brokers List(hds-02.field.hortonworks.com:6667 (id: -2 rack: null), hds-04.field.hortonworks.com:6667 (id: -4 rack: null), hds-01.field.hortonworks.com:6667 (id: -1 rack: null), hds-03.field.hortonworks.com:6667 (id: -3 rack: null)) java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(hds-02.field.hortonworks.com:6667 (id: -2 rack: null), hds-04.field.hortonworks.com:6667 (id: -4 rack: null), hds-01.field.hortonworks.com:6667 (id: -1 rack: null), hds-03.field.hortonworks.com:6667 (id: -3 rack: null)) at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67) at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72) at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125) at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147) at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:320) at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:97) at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:308) at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:76) at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala) <br>
What is wrong here? The user has valid ticket.
I am quite sure the --security-protocol PLAINTEXTSASL should be OK, cause it works on both a console consumer and a producer.
Created 09-20-2017 09:35 PM
It is solved now. Turns out that the ' --security-protocol PLAINTEXTSASL ' is not honored by the kafka-consumer-groups.sh tool.
But it does work if you provide the same through the --command-config flag like:
echo "security.protocol=PLAINTEXTSASL" > /tmp/consgroupcmd.prop /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --new-consumer --describe --group console-consumer-spout1 --command-config /tmp/consgroupcmd.prop
But now, to my suprise it turns out that the kafka-consumer-groups.sh / kafka.admin.ConsumerGroupCommand tool does NOT recognize any consumer group of the --new-consumer type for which offsets are stored on Kafka.
Good old kafka-consumer-offset-checker.sh is still the go-to tool for this (works both on zookeeper and Kafka kept offsets):
/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-offset-checker.sh --zookeeper $ZOOKEEPER_HOSTS --security-protocol PLAINTEXTSASL --topic spout_test --group console-consumer-spout1 KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/2.5.5.0-157/kafka/config/kafka_client_jaas.conf [2017-09-20 21:13:30,168] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$) Group Topic Pid Offset logSize Lag Owner console-consumer-spout1 spout_test 0 36156 68201 32045 none console-consumer-spout1 spout_test 1 0 68198 68198 none console-consumer-spout1 spout_test 2 12053 68198 56145 none console-consumer-spout1 spout_test 3 0 68200 68200 none
It is weird that the deprecation warning recommends to use ConsumerGroupCommand while this does not even work on the new-consumer Kafka offsets:
/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --new-consumer --describe --group console-consumer-spout1 --command-config /tmp/consgroupcmd.prop KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_jaas.conf Consumer group `console-consumer-spout1` does not exist or is rebalancing. [2017-09-20 21:11:04,189] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)
The group 'console-consumer-spout1' does exist as can be seen in the output of the previous command...
Created 09-19-2017 07:28 AM
Hi Jasper,
I don't think you have a problem while logging in.
After skimming through the above link, I guess if there was any error w.r.t. having a valid ticket, you would've got a log in KerberosLogin itself.
What principal are you using?
Can you check contents of kafka_client_jaas.conf. Is it of the form below:
KafkaClient
{ com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
Or you have keytab configuration?
If former, please see kafka_jaas.conf for Client section and kinit with user/keytab mentioned there. Try running the command again as:
/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --security-protocol PLAINTEXTSASL --new-consumer --describe --group spoutconsumer -Djava.security.auth.login.config=
/etc/kafka/kafka_jaas.conf
If above command does not work, try exporting that variable.
Created 09-20-2017 08:44 PM
Thanks for your suggestion. It turns out the kafka-consumer-groups.sh eventually uses :
KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_jaas.conf
The contents of this file is:
KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="kafka" principal="kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM"; }; KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka"; }; Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="zookeeper" principal="kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM"; };
So it does not make a difference what principal is used to kinit. Even when I take the kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM principal I get the same result:
Error while executing consumer group command Request GROUP_COORDINATOR failed on brokers List(dem-mtrn-hds-01.field.hortonworks.com:6667 (id: -1 rack: null), dem-mtrn-hds-04.field.hortonworks.com:6667 (id: -4 rack: null), dem-mtrn-hds-02.field.hortonworks.com:6667 (id: -2 rack: null), dem-mtrn-hds-03.field.hortonworks.com:6667 (id: -3 rack: null)) java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(dem-mtrn-hds-01.field.hortonworks.com:6667 (id: -1 rack: null), dem-mtrn-hds-04.field.hortonworks.com:6667 (id: -4 rack: null), dem-mtrn-hds-02.field.hortonworks.com:6667 (id: -2 rack: null), dem-mtrn-hds-03.field.hortonworks.com:6667 (id: -3 rack: null)) at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67) at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72) at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125) at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147) at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:320) at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:97) at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:308) at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:76) at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Created 09-20-2017 09:35 PM
It is solved now. Turns out that the ' --security-protocol PLAINTEXTSASL ' is not honored by the kafka-consumer-groups.sh tool.
But it does work if you provide the same through the --command-config flag like:
echo "security.protocol=PLAINTEXTSASL" > /tmp/consgroupcmd.prop /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --new-consumer --describe --group console-consumer-spout1 --command-config /tmp/consgroupcmd.prop
But now, to my suprise it turns out that the kafka-consumer-groups.sh / kafka.admin.ConsumerGroupCommand tool does NOT recognize any consumer group of the --new-consumer type for which offsets are stored on Kafka.
Good old kafka-consumer-offset-checker.sh is still the go-to tool for this (works both on zookeeper and Kafka kept offsets):
/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-offset-checker.sh --zookeeper $ZOOKEEPER_HOSTS --security-protocol PLAINTEXTSASL --topic spout_test --group console-consumer-spout1 KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/2.5.5.0-157/kafka/config/kafka_client_jaas.conf [2017-09-20 21:13:30,168] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$) Group Topic Pid Offset logSize Lag Owner console-consumer-spout1 spout_test 0 36156 68201 32045 none console-consumer-spout1 spout_test 1 0 68198 68198 none console-consumer-spout1 spout_test 2 12053 68198 56145 none console-consumer-spout1 spout_test 3 0 68200 68200 none
It is weird that the deprecation warning recommends to use ConsumerGroupCommand while this does not even work on the new-consumer Kafka offsets:
/usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --new-consumer --describe --group console-consumer-spout1 --command-config /tmp/consgroupcmd.prop KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_jaas.conf Consumer group `console-consumer-spout1` does not exist or is rebalancing. [2017-09-20 21:11:04,189] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)
The group 'console-consumer-spout1' does exist as can be seen in the output of the previous command...
Created 09-29-2017 01:58 PM
Reason kafka-consumer-groups.sh returned "Consumergroup`console-consumer-spout1` does not exist or is rebalancing." was that the consumergroup was not active at that moment. If it is not consuming, it is not shown with this command.
In that respect kafka-consumer-offset-checker.sh does a better job as it always returns the offsets, consumer active or not