Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

avatar

As the zookeeper user, after a successful kinit, in a Kerberos enabled cluster,we still cannot invoke zkCli.sh.

The error is:

2016-03-04 12:08:47,596 - ERROR 
[main-SendThread(localhost.localdomain:2181):ZooKeeperSaslClient@384] - 
An error: (java.security.PrivilegedActionException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Server not
 found in Kerberos database (7))]) occurred when evaluating Zookeeper 
Quorum Member's  received SASL token. Zookeeper Client will go to 
AUTH_FAILED state.
1 ACCEPTED SOLUTION

avatar

Thanks for directing me to the right path. Here was the solution

Changed the zookeeper_client_jaas.conf AND zookeeper_client_jaas.conf.j2 template file instead which had useKeyTab=false Change it to the following:

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2
Client {

com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

useTicketCache=false

keyTab="/etc/security/keytabs/zk.service.keytab"

principal="zookeeper/<FQDN>@REALM";

};


However when you restart Ambari, it changes the zookeeper_client_jaas.conf back to the problem, so you HAVE to change to /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2 template file ALSO for Ambari to get correct values.

View solution in original post

7 REPLIES 7

avatar
Master Guru

Do this before trying zkCli.sh

export JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper_jaas.conf"

and make sure jaas.conf is set correctly.

Edit: Sorry, the file is called zookeeper_jaas.conf, I just fixed it.

avatar

Thanks for directing me to the right path. Here was the solution

Changed the zookeeper_client_jaas.conf AND zookeeper_client_jaas.conf.j2 template file instead which had useKeyTab=false Change it to the following:

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2
Client {

com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

useTicketCache=false

keyTab="/etc/security/keytabs/zk.service.keytab"

principal="zookeeper/<FQDN>@REALM";

};


However when you restart Ambari, it changes the zookeeper_client_jaas.conf back to the problem, so you HAVE to change to /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2 template file ALSO for Ambari to get correct values.

avatar
Master Guru

Your solution indicates an unhealthy cluster, and/or erroneous kerberization. Normally, there is no need to touch these files, unless you are installing ZK manually, but you are talking about Ambari. There is also no need for any j2 files. In healthy cluster it's enough to set JVMFLAGS and invoke zkCli.sh. I tested the other day, before filing my answer.

avatar

When using Ambari, I believe that in order to properly execute zkCli.sh, you need to first source the zookeeper-env.sh file.

source /etc/zookeeper/conf/zookeeper-env.sh;
/usr/hdp/current/zookeeper-client/bin/zkCli.sh ....

Modifying zookeeper_jaas.conf to use a service identity is not particularly appropriate for a user-executed task. User executed tasks should be authenticated use the executing user's identity. However I am not an expert with ZooKeeper.

avatar
Contributor

This seems to have worked for me. 1) kinit 2) source the env 3) execute zkCli.sh.

Thanks

avatar
New Contributor

I was unable to start zookeeper-client . after running above steps I am able to log into zookeeper-client and I am able to get the details but while using set command I am facing error like below


Authentication is not valid : /brokers/ids/1001

avatar
Contributor

Hello Satish, this error is not related to the procedure you followed, you need to kinit as kafka's user.