Kafka consumer groups are not getting listed and described with kerberos enabled

[#1]

Hi,

I am using kafka version 0.10.1 on HDP 2.6 with kerberos enabled. When I am trying to get the list of consumer groups and describe group with below commands I am getting error message. Please advise.

[kafka@XXX ~]$ /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --bootstrap-server xxxx:6667,xxxx:6667,xxxx:6667 --list --command-config /tmp/grouprop.properties

[2019-08-13 02:40:56,791] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)

[kafka@XXX ]$ /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server xxxx:6667 --describe --group uat_customer --command-config /tmp/grouprop.properties

Error while executing consumer group command Failed to send request after 5000 ms.

org.apache.kafka.common.errors.TimeoutException: Failed to send request after 5000 ms.

[2019-08-14 09:57:39,346] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)

[kafka@XXX]$ /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server xxxx:6667 --list --command-config /tmp/grouprop.properties

Error while executing consumer group command Request METADATA failed on brokers List(xxx:6667 (id: -1 rack: null))

java.lang.RuntimeException: Request METADATA failed on brokers List(xxxx:6667 (id: -1 rack: null))

at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)

at kafka.admin.AdminClient.findAllBrokers(AdminClient.scala:87)

at kafka.admin.AdminClient.listAllGroups(AdminClient.scala:96)

at kafka.admin.AdminClient.listAllGroupsFlattened(AdminClient.scala:117)

at kafka.admin.AdminClient.listAllConsumerGroupsFlattened(AdminClient.scala:121)

at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.list(ConsumerGroupCommand.scala:304)

at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:66)

at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)

[2019-08-14 10:03:05,877] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)

cat /tmp/grouprop.properties

security.protocol=PLAINTEXTSASL

[#2]

Hi Saurav, 

Can you please enable client DEBUG and see if the security.protocol is passed properly? also, it may help you also to get more detailed information about the timeout issue.

This can be done by editing below file:

/etc/kafka/conf/tools-log4j.properties

Change the following line from:

log4j.rootLogger=WARN, stderr

To

log4j.rootLogger=DEBUG, stderr

After that, run the consumer again and let us know how it goes.

Thanks,

Manuel.

[#3]

Hi @sauravsuman689 

A common issue that people have when using the kafka-consumer-group command line tool is that they do not set it up to communicate over Kerberos like any other Kafka client (i.e. consumers and producers).

The security.protocol output you shared based on the cat command doesn't look right:

cat /tmp/grouprop.properties

security.protocol=PLAINTEXTSASL

This should instead be:

security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka

You can use the same instructions outlined in the following link starting with step number 5:
https://www.cloudera.com/documentation/kafka/latest/topics/kafka_security.html#concept_lcn_4mm_s5

I understand you're using HDP but it should be pretty much the same steps. You will of course just use the same command line tool command you're using as opposed to the consumer command mentioned in the link:

[kafka@XXX ~]$ /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --bootstrap-server xxxx:6667,xxxx:6667,xxxx:6667 --list --command-config /tmp/grouprop.properties

EDIT:

It seems like HDP works a bit differently so your security.protocol parameter aligns with what the HDP platform would expect.