When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

[#1]

When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

[#2 · Accepted solution]

@Avinash C it stores the file on disk. Here is the description. Link

Additionally if you want to look at them, here's info on that. Link

[#3]

Thanks Artem

[#4]

@Avinash C you are welcome please accept the answer to close the thread.

[#5]

@Avinash C

See this line

"The Checkpoint node stores the latest checkpoint in a directory that is structured the same as the NameNode’s directory. This allows the checkpointed image to be always available for reading by the NameNode if necessary. See Import checkpoint."

Exact read on check point https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Checkpoint_...

Please do read the HDFS user guide - 20 -30 minutes read but it will save you lot of time.

[#6]

The link to the Checkpoint Node here is not relevant to HDP or any other modern Hadoop distro AFAIK. The Checkpoint Node provided a way to generate periodic checkpoints of the NameNode metadata. It was an evolution of the SecondaryNameNode. The current architecture is to run NameNode HA using QuorumJournalManager with a redundant pair of NameNodes. In this architecture, whichever NameNode is in standby state also takes responsibility for managing checkpoints as was previously done by the SecondaryNameNode.

[#7]

@Chris Nauroth How about when HA is not enabled?

[#8]

@Neeraj Sabharwal, when HA is not enabled, HDP deployments would rely on the SecondaryNameNode for managing checkpoints.

[#9]

@Chris Nauroth right...So, the above comments are valid?

[#10]

@Neeraj Sabharwal, it is not valid to link to documentation of the CheckpointNode or quote portions of it as a reference. The CheckpointNode and the SecondaryNameNode are similar, but different. The CheckpointNode is not run by HDP or any other distro I've encountered, so discussing it is highly likely to cause confusion. Instead, non-HA deployments would run the SecondaryNameNode.