I'm looking into the possibility of performing an 'online' backup of HDFS metadata without having to take down HDFS or NameNodes and wanted to find out if the following plan is doable or not.
General assumptions:
- Regardless of the solution, we'll never have a full, up-to-date continuous backup of the namespace – we’ll always loose some of the most recent data. It’s not an OLTP system, most of the data can be easily recreated (re-run ETL or processing jobs).
- Normal NN failures are handled by the Standby NN. The goal here is to have a procedure in place for a very unlikely case where both master nodes fail.
- In the case of both NN failures, the NN service can be started up with the most recent image of the namespace we have.
The understanding of how the Name Nodes maintain the namespace, in short, is:
- Standby NN keeps a namespace image in memory based on edits available in a storage ensemble in Journal Nodes.
- Based on pre-conditions (No of transactions or period), standby NN makes a namespace checkpoint and saves a “fsimage_*” to disk.
- Standby NN transfers the fsimage to the primary NN over http.
The understanding is that both NN write fsimages to disk in the following sequence:
- NN writes the namespace to a file “fsimage.ckpt_*” on disk
- NN creates a “fsimage_*.md5” file
- NN moves the file “fsimage.ckpt_*” to “fsimage_.*”
The above means that:
- The most recent namespace image on disk in in a “fsimage_*” file is on the Standby NN.
- Any “fsimage_*” file on disk is finalized and won’t receive more updates.
Based on the above, a proposed, simple procedure what won’t affect the availability of NN is as follows:
- Make sure the Standby NN checkpoints the namespace to “fsimage_” once per hour.
- Backup the most recent “fsimage_*” and “fsimage_*.md5” from the Standby NN periodically. We can try to keep the latest version of the file on another machine in the cluster.
Are there any issues or potential pitfalls with this approach that anyone can see?
