集群故障 -- 生产故障，请求协助

[#1]

联系请登陆zoom：560 656 3972

其中之一的报错如下：

2023-06-21 07:57:03,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@136] - Setting authorizedID: rangerkms 2023-06-21 07:57:03,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1030] - adding SASL authorization for authorizationID: rangerkms 2023-06-21 07:57:03,612 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x488dce1cd620027, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) at java.lang.Thread.run(Thread.java:745) 2023-06-21 07:57:03,612 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1035] - Closed socket connection for client /10.233.68.145:49378 which had sessionid 0x488dce1cd620027 2023-06-21 07:57:03,668 - ERROR [CommitProcessor:4:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1109) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:418) at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:664) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:77)

[#2]

Here is the translation: 

联系请请混合zoom：560 656 3972
One of the errors is as follows:
2023-06-21 07:57:03,547 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@136] - Setting authorizedID: rangerkms 2023-06-21 07:57:03,547 - INFO [NIOServerCxn. .Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1030] - adding SASL authorization for authorizationID: rangerkms 2023-06-21 07:57:03,612 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1030] 2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x488dce1cd620027, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org. .apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) at java.lang.Thread.run(Thread.java:745) 2023-06-21 07:57:03,612 - INFO [NIOServerCnxnFactory.java:0.0. .0.0/0.0.0.0:2181:NIOServerCnxn@1035] - Closed socket connection for client /10.233.68.145:49378 which had sessionid 0x488dce1cd620027 2023-06-21 07:57:03,668 - ERROR [CommitProcessor:4:NIOServerCnxn@178] - UnexpectedException: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache. zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1109) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:1109) 418) at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:664) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:77)

[#3]

@damon,Welcome to our community! To help you get the best possible answer, I have tagged our CDP experts @shehbazk  @vaishaakb @rki_  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#4]

Hi @damon Apart from these CancelledKeyException, are you seeing any other specific warnings or errors in the Zookeeper logs?

[#5 · Accepted solution]

HI @damon The most likely culprit, in this case, would be the Packet length crossing the jute buffer limit. As this is seen for the Kms service, verify if you are getting any warnings such as "Packet len is out of range!"

If you do, then raise the jute buffer for the Zookeeper service as well as for the Kms service ( client ) and restart the respective services to resolve this issue.