Ambari Kerberos Wizard: Zookeeper service won't start

[#1]

We have made it through most of the kerberos wizard but got stuck on the last step where it is attempting to start services. The Zookeeper status check fails and we've found out that zookeeper server is not starting up.

The error in zookeeper.log is:

2016-07-11 14:12:12,565 - INFO  [main:FourLetterWordMain@43] - connecting to localhost 2181
2016-07-11 14:13:34,001 - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Receive timed out
        at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:130)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

I've done some research and found this helpful page about Kerberos errors. Running through the list of possible causes and I am at a loss because we were able to progress through the rest of the wizard OK. All the principals were created by Ambari in Active Directory OK. I can also become the zookeeper user, kinit using zk.service.keytab, and klist perfectly fine.

It seems to me that network issues are the most likely... but shouldn't kinit rule out any firewall or hostname issues with kerberos? Is there a config somewhere I'm missing????

We are using Ambari 2.2.2.0 and HDP 2.3.2.0.

[#2]

@Kit Menke can you verify your KDC is using udp and not tcp?

[#3]

how to verify........?

[#4]

@abhil sam, There are several ways to do this. The easiest is to take a look at the kdc.conf file, usually at /var/kerberos/krb5kdc/kdc.conf. In this file you will see a block that contains something like the following:

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

However, it may not have both properties (kdc_ports, kdc_tcp_ports).

• If it has both, than the KDC is listening on both UDP and TCP sockets on the specified port(s).
• If it only has kdc_ports, than it is listening on UDP only.
• If it only has kdc_tcp_ports, then it is listening on TCP only.

Another way is to use the netcat (nc) utility:

TCP: nc -vz -t hostname 88
UDP: nc -vz -u hostname 88

You seem to need to use the actual hostname or FQDN for the UDP socket test to work. I tried localhost and it didn't work.

Examples:

[root@my_hostname ~]# nc -vz -u my_hostname 88
Connection to my_hostname 88 port [udp/kerberos] succeeded!


[root@my_hostname ~]# nc -vz -t my_hostname 88
Connection to my_hostname 88 port [tcp/kerberos] succeeded!

[#5]

when i try for tcp port i am getting connection refused

for udp it is ok...

what could be the reason

[#6 · Accepted solution]

Hey @Kit Menke

Have you tried turning off UDP? If not, add the property and value:

udp_preference_limit = 1 

to your /etc/krb5.conf and see if that solves your problem.

If you have Ambari managing the krb5.conf, go to Kerberos -> Advanced krb5.conf -> krb5-conf template and add the property to the [libdefaults] section. Save the changes and let Ambari push out the changes to the hosts.

If you are managing the /etc/krb5.conf file, then you will have to add the property and push out changes yourself.

[#7]

Thanks @Terry Padgett! This worked and we were able to start zookeeper after adding this entry in ambari. Looks like we'll need to follow up with networking to see about opening up UDP.