Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

SOLVED Go to solution
Highlighted

zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

As the zookeeper user, after a successful kinit, in a Kerberos enabled cluster,we still cannot invoke zkCli.sh.

The error is:

2016-03-04 12:08:47,596 - ERROR 
[main-SendThread(localhost.localdomain:2181):ZooKeeperSaslClient@384] - 
An error: (java.security.PrivilegedActionException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Server not
 found in Kerberos database (7))]) occurred when evaluating Zookeeper 
Quorum Member's  received SASL token. Zookeeper Client will go to 
AUTH_FAILED state.
1 ACCEPTED SOLUTION

Accepted Solutions

Re: zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

Thanks for directing me to the right path. Here was the solution

Changed the zookeeper_client_jaas.conf AND zookeeper_client_jaas.conf.j2 template file instead which had useKeyTab=false Change it to the following:

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2
Client {

com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

useTicketCache=false

keyTab="/etc/security/keytabs/zk.service.keytab"

principal="zookeeper/<FQDN>@REALM";

};


However when you restart Ambari, it changes the zookeeper_client_jaas.conf back to the problem, so you HAVE to change to /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2 template file ALSO for Ambari to get correct values.

7 REPLIES 7

Re: zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

Do this before trying zkCli.sh

export JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper_jaas.conf"

and make sure jaas.conf is set correctly.

Edit: Sorry, the file is called zookeeper_jaas.conf, I just fixed it.

Re: zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

Thanks for directing me to the right path. Here was the solution

Changed the zookeeper_client_jaas.conf AND zookeeper_client_jaas.conf.j2 template file instead which had useKeyTab=false Change it to the following:

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf

# vi /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2
Client {

com.sun.security.auth.module.Krb5LoginModule required

useKeyTab=true

storeKey=true

useTicketCache=false

keyTab="/etc/security/keytabs/zk.service.keytab"

principal="zookeeper/<FQDN>@REALM";

};


However when you restart Ambari, it changes the zookeeper_client_jaas.conf back to the problem, so you HAVE to change to /etc/zookeeper/conf/zookeeper_client_jaas.conf.j2 template file ALSO for Ambari to get correct values.

Re: zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

Your solution indicates an unhealthy cluster, and/or erroneous kerberization. Normally, there is no need to touch these files, unless you are installing ZK manually, but you are talking about Ambari. There is also no need for any j2 files. In healthy cluster it's enough to set JVMFLAGS and invoke zkCli.sh. I tested the other day, before filing my answer.

Re: zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

When using Ambari, I believe that in order to properly execute zkCli.sh, you need to first source the zookeeper-env.sh file.

source /etc/zookeeper/conf/zookeeper-env.sh;
/usr/hdp/current/zookeeper-client/bin/zkCli.sh ....

Modifying zookeeper_jaas.conf to use a service identity is not particularly appropriate for a user-executed task. User executed tasks should be authenticated use the executing user's identity. However I am not an expert with ZooKeeper.

Re: zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

Contributor

This seems to have worked for me. 1) kinit 2) source the env 3) execute zkCli.sh.

Thanks

Re: zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

New Contributor

I was unable to start zookeeper-client . after running above steps I am able to log into zookeeper-client and I am able to get the details but while using set command I am facing error like below


Authentication is not valid : /brokers/ids/1001

Re: zkCli cannot execute in a Kerberos cluster - zookeeper Client will go to AUTH_FAILED state

Contributor

Hello Satish, this error is not related to the procedure you followed, you need to kinit as kafka's user.