ERROR org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer: Failed to authenticate using SASL

[#1]

Hello,

Cluster has been Kerberized (LDAP / AD / Kerberos) and I have errors when I try to start the cluster. Zookeeper service start with following error:

2022-07-01 13:24:14,341 ERROR org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer: Failed to authenticate using SASL

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt

AP-REQ - RC4 with HMAC)]

        at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199)

        at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.authenticate(SaslQuorumAuthServer.java:99)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:563)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:487)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReceiverThread.run(QuorumCnxManager.java:523)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

        at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - RC4 with HMAC)

        at java.security.jgss/sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:859)

        at java.security.jgss/sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:361)

        at java.security.jgss/sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:303)

        at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167)

        ... 7 more

Caused by: KrbException: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - RC4 with HMAC

        at java.security.jgss/sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:278)

        at java.security.jgss/sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149)

        at java.security.jgss/sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:139)

        at java.security.jgss/sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:832)

        ... 10 more

2022-07-01 13:24:14,341 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager: Exception handling connection, addr: /x.x.x.222:35604, closing server connection

 

2022-07-01 13:24:14,476 INFO org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner: QuorumLearner will use GSSAPI as SASL mechanism.

2022-07-01 13:24:14,476 INFO org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner: QuorumLearner will use GSSAPI as SASL mechanism.

2022-07-01 13:24:14,477 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager: Exception while connecting, id: [2, FQDN/x.x.x.221:4181], addr: {}, closing learner connection

javax.security.sasl.SaslException: Authentication failed against server addr: FQDN/x.x.x.221:4181

        at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.authenticate(SaslQuorumAuthLearner.java:126)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.startConnection(QuorumCnxManager.java:442)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:353)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:402)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

        at java.base/java.lang.Thread.run(Thread.java:829)

2022-07-01 13:24:14,478 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager: Exception while connecting, id: [3, FQDN/x.x.x.222:4181], addr: {}, closing learner connection

javax.security.sasl.SaslException: Authentication failed against server addr: FQDN/x.x.x.222:4181

        at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.authenticate(SaslQuorumAuthLearner.java:126)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.startConnection(QuorumCnxManager.java:442)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:353)

        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:402)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

        at java.base/java.lang.Thread.run(Thread.java:829)

2022-07-01 13:24:14,906 WARN org.apache.zookeeper.server.NettyServerCnxn: Closing connection to /x.x.x.220:60416

java.io.IOException: ZK down

        at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:474)

        at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360)

        at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:266)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)

        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)

        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)

        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)

        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)

        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.base/java.lang.Thread.run(Thread.java:829)

2022-07-01 13:24:18,478 WARN org.apache.zookeeper.server.NettyServerCnxn: Closing connection to /x.x.x.220:60456

java.io.IOException: ZK down

        at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:474)

        at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360)

        at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:266)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)

        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)

        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)

        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)

        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)

        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.base/java.lang.Thread.run(Thread.java:829)

This line confuses me because I'm using different encryption type: aes256-cts-hmac-sha1-96

Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - RC4 with HMAC)

HDFS and other services failed to start. Any advice would be appreciated. 

[#2 · Accepted solution]

@stale ,

It looks like a mismatch in the encryption types in your krb5.conf and the AD is causing this. Do check the below 2 Cloudera articles to see if that helps resolving this issue.

https://my.cloudera.com/knowledge/ERRORquotCaused-by-GSSException-Failure-unspecified-at-GSS-API?id=...

https://my.cloudera.com/knowledge/ErrorquotCaused-by-Failure-unspecified-at-GSS-API-level?id=273436

[#3]

I am facing the same error, however our AD supports aes encryption type.

klist -kte zookeeper.keytab
Keytab name: FILE:zookeeper.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
1 06/09/2023 13:03:47 zookeeper/hostname@REALM (aes256-cts-hmac-sha1-96)

klist -Aef
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: zookeeper/host@REALM

Valid starting Expires Service principal
06/09/2023 15:52:48 06/10/2023 01:52:48 krbtgt/REALM@REALM
renew until 06/16/2023 15:52:48, Flags: FRIA
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

[#4]

@Airtel, Welcome to our community! To help you get the best possible answer, I have tagged in our CDP experts @rki_ @vaishaakb  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.