Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

Solved Go to solution
Highlighted

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

The prior answer does not actually answer the original stated question.

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

Yes, old fsimage files get deleted. However, a certain number of prior fsimage files will be retained. The exact number to retain is controlled by configuration property dfs.namenode.num.checkpoints.retained in hdfs-site.xml. If unspecified, then the default value is 2.

<property>
  <name>dfs.namenode.num.checkpoints.retained</name>
  <value>2</value>
  <description>The number of image checkpoint files (fsimage_*) that will be retained by
  the NameNode and Secondary NameNode in their storage directories. All edit
  logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained
  checkpoint will also be retained.
  </description>
</property>

The reason for retaining a few prior fsimage files is that it can be useful for post-mortem troubleshooting or in some disastrous cases as a way to restore a cluster to a prior state. (However, restoring an old fsimage will cause loss of all data that was saved after that checkpoint, so this is not standard operating procedure.)

View solution in original post

10 REPLIES 10
Highlighted

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

@Avinash C

See this description

"The secondary NameNode merges the fsimage and the edits log files periodically and keeps edits log size within a limit. It is usually run on a different machine than the primary NameNode since its memory requirements are on the same order as the primary NameNode."

The Checkpoint node periodically creates checkpoints of the namespace. It downloads fsimage and edits from the active NameNode, merges them locally, and uploads the new image back to the active NameNode. The Checkpoint node usually runs on a different machine than the NameNode since its memory requirements are on the same order as the NameNode. The Checkpoint node is started by bin/hdfs namenode -checkpoint on the node specified in the configuration file.

The location of the Checkpoint (or Backup) node and its accompanying web interface are configured via the dfs.namenode.backup.address and dfs.namenode.backup.http-addressconfiguration variables.

The start of the checkpoint process on the Checkpoint node is controlled by two configuration parameters.

  • dfs.namenode.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints
  • dfs.namenode.checkpoint.txns, set to 1 million by default, defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not been reached.

The Checkpoint node stores the latest checkpoint in a directory that is structured the same as the NameNode's directory. This allows the checkpointed image to be always available for reading by the NameNode if necessary. See Import checkpoint.

Highlighted

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

@Avinash C Details link

Highlighted

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

Thanks Neeraj for details explanation of check-pointing. However I just wanted to know what happens with the old fsimage file stored in NameNode i.e. does it get replaced with new fsimage or does it maintain the old and new fsimage as well?

Highlighted

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

@Avinash C

Yes , new fsimage

old fsimage + edit logs = new fsimage

Highlighted

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

See this explanation.

When the NameNode starts up, it reads the FsImage and EditLog from disk, applies all the transactions from the EditLog to the in-memory representation of the FsImage, and flushes out this new version into a new FsImage on disk. It can then truncate the old EditLog because its transactions have been applied to the persistent FsImage. This process is called a checkpoint

Highlighted

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

Thanks Neeraj, appreciate your response

Highlighted

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

This answer quotes a lot of Apache documentation on the CheckpointNode. The Checkpoint Node has never been used in HDP deployments, and I have not personally encountered it in other distros either. A modern HA deployment will use NameNode HA using QuorumJournalManager. A non-HA deployment will run the SecondaryNameNode, which is slightly different from the CheckpointNode.

Highlighted

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

The prior answer does not actually answer the original stated question.

When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

Yes, old fsimage files get deleted. However, a certain number of prior fsimage files will be retained. The exact number to retain is controlled by configuration property dfs.namenode.num.checkpoints.retained in hdfs-site.xml. If unspecified, then the default value is 2.

<property>
  <name>dfs.namenode.num.checkpoints.retained</name>
  <value>2</value>
  <description>The number of image checkpoint files (fsimage_*) that will be retained by
  the NameNode and Secondary NameNode in their storage directories. All edit
  logs (stored on edits_* files) necessary to recover an up-to-date namespace from the oldest retained
  checkpoint will also be retained.
  </description>
</property>

The reason for retaining a few prior fsimage files is that it can be useful for post-mortem troubleshooting or in some disastrous cases as a way to restore a cluster to a prior state. (However, restoring an old fsimage will cause loss of all data that was saved after that checkpoint, so this is not standard operating procedure.)

View solution in original post

Re: When Secondary NameNode performs checkpoint i.e. once it writes the updated fsimage to NameNode, does the old fsimage file gets deleted?

Contributor

Hi Chris, my cluster was hacked and the HDFS data was deleted (includ /user/ and the trash files). I can see in /hadoop/hdfs/namenode the fsimage_ file before the deletes were applied. Could you explain how I would go about reverting to the older fsimage_ file?

Don't have an account?
Coming from Hortonworks? Activate your account here