Hello,
Cluster has been Kerberized (LDAP / AD / Kerberos) and I have errors when I try to start the cluster. Zookeeper service start with following error:
2022-07-01 13:24:14,341 ERROR org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer: Failed to authenticate using SASL
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt
AP-REQ - RC4 with HMAC)]
at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199)
at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.authenticate(SaslQuorumAuthServer.java:99)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:563)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:487)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReceiverThread.run(QuorumCnxManager.java:523)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - RC4 with HMAC)
at java.security.jgss/sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:859)
at java.security.jgss/sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:361)
at java.security.jgss/sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:303)
at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167)
... 7 more
Caused by: KrbException: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - RC4 with HMAC
at java.security.jgss/sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:278)
at java.security.jgss/sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149)
at java.security.jgss/sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:139)
at java.security.jgss/sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:832)
... 10 more
2022-07-01 13:24:14,341 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager: Exception handling connection, addr: /x.x.x.222:35604, closing server connection
2022-07-01 13:24:14,476 INFO org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner: QuorumLearner will use GSSAPI as SASL mechanism.
2022-07-01 13:24:14,476 INFO org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner: QuorumLearner will use GSSAPI as SASL mechanism.
2022-07-01 13:24:14,477 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager: Exception while connecting, id: [2, FQDN/x.x.x.221:4181], addr: {}, closing learner connection
javax.security.sasl.SaslException: Authentication failed against server addr: FQDN/x.x.x.221:4181
at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.authenticate(SaslQuorumAuthLearner.java:126)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.startConnection(QuorumCnxManager.java:442)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:353)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:402)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-07-01 13:24:14,478 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager: Exception while connecting, id: [3, FQDN/x.x.x.222:4181], addr: {}, closing learner connection
javax.security.sasl.SaslException: Authentication failed against server addr: FQDN/x.x.x.222:4181
at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.authenticate(SaslQuorumAuthLearner.java:126)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.startConnection(QuorumCnxManager.java:442)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:353)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:402)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-07-01 13:24:14,906 WARN org.apache.zookeeper.server.NettyServerCnxn: Closing connection to /x.x.x.220:60416
java.io.IOException: ZK down
at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:474)
at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360)
at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-07-01 13:24:18,478 WARN org.apache.zookeeper.server.NettyServerCnxn: Closing connection to /x.x.x.220:60456
java.io.IOException: ZK down
at org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:474)
at org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360)
at org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
This line confuses me because I'm using different encryption type: aes256-cts-hmac-sha1-96
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - RC4 with HMAC)
HDFS and other services failed to start. Any advice would be appreciated.
