zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

[#1]

As the zookeeper user, after a successful kinit, in a Kerberos enabled cluster,we still cannot invoke zkCli.sh.

The error is:

2016-03-04 12:08:47,596 - ERROR 
[main-SendThread(localhost.localdomain:2181):ZooKeeperSaslClient@384] - 
An error: (java.security.PrivilegedActionException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Server not
 found in Kerberos database (7))]) occurred when evaluating Zookeeper 
Quorum Member's  received SASL token. Zookeeper Client will go to 
AUTH_FAILED state.

[#2]

Do this before trying zkCli.sh

export JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper_jaas.conf"

and make sure jaas.conf is set correctly.

Edit: Sorry, the file is called zookeeper_jaas.conf, I just fixed it.

[#3 · Accepted solution]

Thanks for directing me to the right path. Here was the solution

Changed the zookeeper_client_jaas.conf AND zookeeper_client_jaas.conf.j2 template file instead which had useKeyTab=false Change it to the following:

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2

Client {

com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

useTicketCache=false

keyTab="/etc/security/keytabs/zk.service.keytab"

principal="zookeeper/<FQDN>@REALM";

};

However when you restart Ambari, it changes the zookeeper_client_jaas.conf back to the problem, so you HAVE to change to /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2 template file ALSO for Ambari to get correct values.

[#4]

Your solution indicates an unhealthy cluster, and/or erroneous kerberization. Normally, there is no need to touch these files, unless you are installing ZK manually, but you are talking about Ambari. There is also no need for any j2 files. In healthy cluster it's enough to set JVMFLAGS and invoke zkCli.sh. I tested the other day, before filing my answer.

[#5]

When using Ambari, I believe that in order to properly execute zkCli.sh, you need to first source the zookeeper-env.sh file.

source /etc/zookeeper/conf/zookeeper-env.sh;
/usr/hdp/current/zookeeper-client/bin/zkCli.sh ....

Modifying zookeeper_jaas.conf to use a service identity is not particularly appropriate for a user-executed task. User executed tasks should be authenticated use the executing user's identity. However I am not an expert with ZooKeeper.

[#6]

This seems to have worked for me. 1) kinit 2) source the env 3) execute zkCli.sh.

Thanks

[#7]

I was unable to start zookeeper-client . after running above steps I am able to log into zookeeper-client and I am able to get the details but while using set command I am facing error like below

Authentication is not valid : /brokers/ids/1001

[#8]

Hello Satish, this error is not related to the procedure you followed, you need to kinit as kafka's user.