Currently our zookeeper dataDir is at `/dfs/1/hadoop/zookeeper/` but unfortunately, `/dfs/1/` is the HDFS disk mount. Hence in current scenario, for us it is not possible to swap disks for HDFS as zookeeper is also using it.
We wanted to move zookeeper dataDir to some other place like `/usr/lib/zookeeper` but I am not quite sure of what steps needs to be taken. Here's what I think should work.
Here what I'm unsure of is if copying the data is the correct way to do this. We do not have staging cluster hence seeking help from community :)
If you have 3 or more zookeeper servers, then you could carry out these steps on each Zookeeper one by one, in a rolling fashion, thereby keeping the zookeeper quorum intact. Otherwise, the mentioned steps are fine.
Copy all files inside current dataDir (myid , version-2) to the new directory, update 'dataDir' and 'dataLogDir' (if separately configured) properties inside zoo.cfg, set the directory ownership (recursive) to zk service user and restart the zookeeper server. During the startup, zookeeper loads the latest 'snapshot' file and replays the transaction file to load the state. Follower zookeepers also syncs with the leader for the current state.
Thanks @rmaruthiyodan , So let me confirm the steps:
yes we have 3 zookeeper nodes. I wanted to ask if above steps can be executed while HBase is running. (they should be)
Yes @sanket patel, You can proceed the same step.
Zookeeper will sync the latest snapshot & logs from remaining quorums so you can skip step3 or copy as your own interest but make sure you provide little time for sync before restart another zk servers.
Thanks all for help! I carried out steps as I mentioned in question. Please include `chown -R` operation too before starting services as mentioned by @rmaruthiyodan
We did it with approx 5min of downtime though, If anyone else carries this operation out without downtime/in rolling fashion, please let community know.
Hi, Please find the below steps for moving zookeeper data directory.