Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Zookeeper service stops on its own

Zookeeper service stops on its own

New Contributor

I run a small cluster on virtual machines for testing and learning. At one time my Windows host crashed and I was forced to hard reset the computer. Afterward, zookeeper server on one host would stop on its own within one minute after starting. The log file is as follows:

java.lang.RuntimeException: Unable to run quorum server

at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:531)

at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:473)

at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)

at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)

at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

Caused by: java.io.IOException: Input/output error

at java.io.FileInputStream.readBytes(Native Method)

at java.io.FileInputStream.read(FileInputStream.java:255)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

at java.io.BufferedInputStream.read(BufferedInputStream.java:265)

at java.io.FilterInputStream.read(FilterInputStream.java:83)

at org.apache.zookeeper.server.persistence.FileTxnLog$PositionInputStream.read(FileTxnLog.java:443)

at java.io.DataInputStream.readInt(DataInputStream.java:387)

at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)

at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:576)

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:595)

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:561)

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.init(FileTxnLog.java:543)

at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.<init>(FileTxnLog.java:522)

at org.apache.zookeeper.server.persistence.FileTxnLog.read(FileTxnLog.java:354)

at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:132)

at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)

at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:483)

... 4 more

3 REPLIES 3

Re: Zookeeper service stops on its own

Rising Star

Hello,

@Kevin Gao, can you check if you can still write to the hdfs? We have had similar issues happening with hard resets where the docker FS got corrupted.

Highlighted

Re: Zookeeper service stops on its own

Contributor
@Kevin Gao

It appears to be a I/O error . Please check if there is enough storage .

Also validate the permissions as well

Re: Zookeeper service stops on its own

New Contributor

Hi, I'm facing the same problem, since I was working on a test environment I had the possibility to uninstall and reinstall the zookeeper server on the node (hoping to overtake the permission problem), but the problem remains.

I've already checked the availability of enough space on the devices, any other suggestion?

Env: RHEL 7.4 (Maipo), HDP 2.6.3.0-235 (Zookeeper 3.4.6).