Zookeeper server is not coming up

[#1]

Hello Experts,

One of the zookeeper server out of three has gone down and after applying various workaround it is still not coming up. We have three in total including one as the zookeeper leader and other two are followers.

Error:

INFO org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot /var/lib/zookeeper/version-2/snapshot.6801207d537
2022-11-05 13:54:57,287 ERROR org.apache.zookeeper.server.quorum.QuorumPeer: Unable to load database on disk
java.io.EOFException

Tried restarting multiple times all zookeeper servers and also just restarting leader and then followers but issue still persist.

Also moved the snapshot to tmp by running command

mv /var//lib/zookeeper/version-2/snapshot.6801207d537 /tmp

and then restarted but not coming up.

Please advise what else can be done in this scenario.

[#2]

Can you check with this solution?

https://community.cloudera.com/t5/Support-Questions/zookeeper-server-Unable-to-load-database-on-disk...

[#3]

Yes I had already applied this workaround but it didn't work. 

[#4]

Hi Hanzala,

Thank you for raising concern in community.

If the snapshot is moved then the info will not the show the below error::
--------------

 Reading snapshot /var/lib/zookeeper/version-2/snapshot.6801207d537

There might some disk issue which cause the snapshot to be not end the file properly

from the problematic zookeeper

1.move the ZK's Data Directory to backup Directory
2. Copy the transition logs and version-2/* from any running zookeeper to problematic zookeeper node
3. Then try starting problematic ZK node again

And please update us the new log