When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

[#1]

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

[#2]

@Avinash C

See this description

"The secondary NameNode merges the fsimage and the edits log files periodically and keeps edits log size within a limit. It is usually run on a different machine than the primary NameNode since its memory requirements are on the same order as the primary NameNode."

The Checkpoint node periodically creates checkpoints of the namespace. It downloads fsimage and edits from the active NameNode, merges them locally, and uploads the new image back to the active NameNode. The Checkpoint node usually runs on a different machine than the NameNode since its memory requirements are on the same order as the NameNode. The Checkpoint node is started by bin/hdfs namenode -checkpoint on the node specified in the configuration file.

The location of the Checkpoint (or Backup) node and its accompanying web interface are configured via the dfs.namenode.backup.address and dfs.namenode.backup.http-addressconfiguration variables.

The start of the checkpoint process on the Checkpoint node is controlled by two configuration parameters.

• dfs.namenode.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints
• dfs.namenode.checkpoint.txns, set to 1 million by default, defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not been reached.

The Checkpoint node stores the latest checkpoint in a directory that is structured the same as the NameNode's directory. This allows the checkpointed image to be always available for reading by the NameNode if necessary. See Import checkpoint.

[#3]

@Avinash C Details link

[#4]

Thanks Neeraj for details explanation of check-pointing. However I just wanted to know what happens with the old fsimage file stored in NameNode i.e. does it get replaced with new fsimage or does it maintain the old and new fsimage as well?

[#5]

@Avinash C

Yes , new fsimage

old fsimage + edit logs = new fsimage

[#6]

See this explanation.

When the NameNode starts up, it reads the FsImage and EditLog from disk, applies all the transactions from the EditLog to the in-memory representation of the FsImage, and flushes out this new version into a new FsImage on disk. It can then truncate the old EditLog because its transactions have been applied to the persistent FsImage. This process is called a checkpoint

[#7]

Thanks Neeraj, appreciate your response

[#8]

This answer quotes a lot of Apache documentation on the CheckpointNode. The Checkpoint Node has never been used in HDP deployments, and I have not personally encountered it in other distros either. A modern HA deployment will use NameNode HA using QuorumJournalManager. A non-HA deployment will run the SecondaryNameNode, which is slightly different from the CheckpointNode.

[#9 · Accepted solution]

The prior answer does not actually answer the original stated question.

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

Yes, old fsimage files get deleted. However, a certain number of prior fsimage files will be retained. The exact number to retain is controlled by configuration property dfs.namenode.num.checkpoints.retained in hdfs-site.xml. If unspecified, then the default value is 2.

<property>
  <name>dfs.namenode.num.checkpoints.retained</name>
  <value>2</value>
  <description>The number of image checkpoint files (fsimage_*) that will be retained by
  the NameNode and Secondary NameNode in their storage directories. All edit
  logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained
  checkpoint will also be retained.
  </description>
</property>

The reason for retaining a few prior fsimage files is that it can be useful for post-mortem troubleshooting or in some disastrous cases as a way to restore a cluster to a prior state. (However, restoring an old fsimage will cause loss of all data that was saved after that checkpoint, so this is not standard operating procedure.)

[#10]

Hi Chris, my cluster was hacked and the HDFS data was deleted (includ /user/ and the trash files). I can see in /hadoop/hdfs/namenode the fsimage_ file before the deletes were applied. Could you explain how I would go about reverting to the older fsimage_ file?