question Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted? in Archives of Support Questions (Read Only)

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

clouderazone — Sat, 30 Jan 2016 20:43:11 GMT

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

nsabharwal — Sat, 30 Jan 2016 20:51:36 GMT

@Avinash C

See this description

"The secondary NameNode merges the fsimage and the edits log files periodically and keeps edits log size within a limit. It is usually run on a different machine than the primary NameNode since its memory requirements are on the same order as the primary NameNode."

The Checkpoint node periodically creates checkpoints of the namespace. It downloads fsimage and edits from the active NameNode, merges them locally, and uploads the new image back to the active NameNode. The Checkpoint node usually runs on a different machine than the NameNode since its memory requirements are on the same order as the NameNode. The Checkpoint node is started by bin/hdfs namenode -checkpoint on the node specified in the configuration file.

The location of the Checkpoint (or Backup) node and its accompanying web interface are configured via the dfs.namenode.backup.address and dfs.namenode.backup.http-addressconfiguration variables.

The start of the checkpoint process on the Checkpoint node is controlled by two configuration parameters.

dfs.namenode.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints
dfs.namenode.checkpoint.txns, set to 1 million by default, defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not been reached.

The Checkpoint node stores the latest checkpoint in a directory that is structured the same as the NameNode's directory. This allows the checkpointed image to be always available for reading by the NameNode if necessary. See Import checkpoint.

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

nsabharwal — Sat, 30 Jan 2016 20:54:33 GMT

@Avinash C Details link

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

clouderazone — Sat, 30 Jan 2016 21:02:13 GMT

Thanks Neeraj for details explanation of check-pointing. However I just wanted to know what happens with the old fsimage file stored in NameNode i.e. does it get replaced with new fsimage or does it maintain the old and new fsimage as well?

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

nsabharwal — Sat, 30 Jan 2016 21:15:36 GMT

@Avinash C

Yes , new fsimage

old fsimage + edit logs = new fsimage

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

nsabharwal — Sat, 30 Jan 2016 21:16:43 GMT

See this explanation.

When the NameNode starts up, it reads the FsImage and EditLog from disk, applies all the transactions from the EditLog to the in-memory representation of the FsImage, and flushes out this new version into a new FsImage on disk. It can then truncate the old EditLog because its transactions have been applied to the persistent FsImage. This process is called a checkpoint

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

clouderazone — Sat, 30 Jan 2016 21:42:22 GMT

Thanks Neeraj, appreciate your response

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

cnauroth — Thu, 04 Feb 2016 02:00:01 GMT

This answer quotes a lot of Apache documentation on the CheckpointNode. The Checkpoint Node has never been used in HDP deployments, and I have not personally encountered it in other distros either. A modern HA deployment will use NameNode HA using QuorumJournalManager. A non-HA deployment will run the SecondaryNameNode, which is slightly different from the CheckpointNode.

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

cnauroth — Thu, 04 Feb 2016 02:04:47 GMT

The prior answer does not actually answer the original stated question.

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

Yes, old fsimage files get deleted. However, a certain number of prior fsimage files will be retained. The exact number to retain is controlled by configuration property dfs.namenode.num.checkpoints.retained in hdfs-site.xml. If unspecified, then the default value is 2.

<property>
  <name>dfs.namenode.num.checkpoints.retained</name>
  <value>2</value>
  <description>The number of image checkpoint files (fsimage_*) that will be retained by
  the NameNode and Secondary NameNode in their storage directories. All edit
  logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained
  checkpoint will also be retained.
  </description>
</property>

The reason for retaining a few prior fsimage files is that it can be useful for post-mortem troubleshooting or in some disastrous cases as a way to restore a cluster to a prior state. (However, restoring an old fsimage will cause loss of all data that was saved after that checkpoint, so this is not standard operating procedure.)

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

PARTOMIA — Wed, 27 Apr 2016 17:34:27 GMT

@Avinash C

The process of checkpointing is to merge the old fsimage and edits logs to create a checkpoint file. This file is named something like - "fsimage.ckpt_*".

If the checkpointing is successful, then the fsimage.ckpt_* file gets renamed. Internally, the "fsimage.ckpt_*" file is first validated and verified by namenode and then these are renamed to new "fsimage".

If on the other hand, the "fsimage.ckpt_*" are found to be invalid by namenode, then these are not renamed and they stay in the namenode directory. This can later be used for understanding the reason for file to be invalid.

An example of "fsimage.ckpt_*" getting invalid can be, if the namenode got killed while checkpointing was in progress, before renaming the "fsimage.ckpt_*" file to actual fsimage file. This will leave the checkpointing as incomplete and on next NN start ( or checkpoinintg ), it will start the checkpointing again by loading the previous fsimage and applying rest of the edits.

It will not use the last "fsimage.ckpt_*" file.

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

mph — Sun, 08 Jan 2017 20:01:48 GMT

Hi Chris, my cluster was hacked and the HDFS data was deleted (includ /user/ and the trash files). I can see in /hadoop/hdfs/namenode the fsimage_ file before the deletes were applied. Could you explain how I would go about reverting to the older fsimage_ file?