Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Problems with Kafka Scripts after enabled Kerberos

Solved Go to solution
Highlighted

Problems with Kafka Scripts after enabled Kerberos

Expert Contributor

After enabled Kerberos using Ambari wizard, Kafka scripts does not work. Is there any additional configurations to make it work? I am using HDP 2.5.3.

$ kinit
$ ./kafka-topics.sh --zookeeper localhost:2181 --create --topic foo --partitions 1 --replication-factor 1
[2017-01-20 11:54:59,482] WARN Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2017-01-20 11:54:59,484] WARN SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
Exception in thread "main" org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure
	at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:946)
	at org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:923)
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1230)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
	at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:75)
	at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Problems with Kafka Scripts after enabled Kerberos

Expert Contributor

Solved by the below workaround. This looks like a bug in kafka-topics.sh.

1. Add KAFKA_CLIENT_KERBEROS_PARAMS before executing actual TopicCommand if running in a Kerberos enabled cluster.

$ cat kafka-topics.sh
# check if kafka_jaas.conf in config , only enable client_kerberos_params in secure mode.
KAFKA_HOME="$(dirname $(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ))"
KAFKA_JAAS_CONF=$KAFKA_HOME/config/kafka_jaas.conf
if [ -f $KAFKA_JAAS_CONF ]; then
    export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=$KAFKA_HOME/config/kafka_client_jaas.conf"
fi

exec $(dirname $0)/kafka-run-class.sh kafka.admin.TopicCommand "$@"

2. Use Zookeeper server FQDN instead of localhost in command line.

$ kinit
$ ./kafka-topics.sh --zookeeper ip-10-0-0-149.ap-northeast-1.compute.internal:2181 --create --topic foo --partitions 1 --replication-factor 1

Created topic "foo".
3 REPLIES 3

Re: Problems with Kafka Scripts after enabled Kerberos

Expert Contributor

Also tried the below, didn't work...

$ export JVMFLAGS="-Djava.security.auth.login.config=/etc/kafka/conf/kafka_client_jaas.conf"

$ ./kafka-topics.sh --zookeeper localhost:2181 --create --topic foo --partitions 1 --replication-factor 1

Re: Problems with Kafka Scripts after enabled Kerberos

Expert Contributor

Solved by the below workaround. This looks like a bug in kafka-topics.sh.

1. Add KAFKA_CLIENT_KERBEROS_PARAMS before executing actual TopicCommand if running in a Kerberos enabled cluster.

$ cat kafka-topics.sh
# check if kafka_jaas.conf in config , only enable client_kerberos_params in secure mode.
KAFKA_HOME="$(dirname $(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ))"
KAFKA_JAAS_CONF=$KAFKA_HOME/config/kafka_jaas.conf
if [ -f $KAFKA_JAAS_CONF ]; then
    export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=$KAFKA_HOME/config/kafka_client_jaas.conf"
fi

exec $(dirname $0)/kafka-run-class.sh kafka.admin.TopicCommand "$@"

2. Use Zookeeper server FQDN instead of localhost in command line.

$ kinit
$ ./kafka-topics.sh --zookeeper ip-10-0-0-149.ap-northeast-1.compute.internal:2181 --create --topic foo --partitions 1 --replication-factor 1

Created topic "foo".

Re: Problems with Kafka Scripts after enabled Kerberos

Expert Contributor

@yjiang

That helped me aswell. Thank you!

Don't have an account?
Coming from Hortonworks? Activate your account here