question Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk? in Archives of Support Questions (Read Only)

When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

clouderazone — Sat, 30 Jan 2016 13:03:56 GMT

When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

aervits — Sat, 30 Jan 2016 20:26:22 GMT

@Avinash C it stores the file on disk. Here is the description. Link

Additionally if you want to look at them, here's info on that. Link

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

clouderazone — Sat, 30 Jan 2016 20:32:41 GMT

Thanks Artem

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

nsabharwal — Sat, 30 Jan 2016 21:09:42 GMT

@Avinash C

See this line

"The Checkpoint node stores the latest checkpoint in a directory that is structured the same as the NameNode’s directory. This allows the checkpointed image to be always available for reading by the NameNode if necessary. See Import checkpoint."

Exact read on check point https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Checkpoint_Node

Please do read the HDFS user guide - 20 -30 minutes read but it will save you lot of time.

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

aervits — Sat, 30 Jan 2016 21:21:41 GMT

@Avinash C you are welcome please accept the answer to close the thread.

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

cnauroth — Thu, 04 Feb 2016 01:51:40 GMT

The link to the Checkpoint Node here is not relevant to HDP or any other modern Hadoop distro AFAIK. The Checkpoint Node provided a way to generate periodic checkpoints of the NameNode metadata. It was an evolution of the SecondaryNameNode. The current architecture is to run NameNode HA using QuorumJournalManager with a redundant pair of NameNodes. In this architecture, whichever NameNode is in standby state also takes responsibility for managing checkpoints as was previously done by the SecondaryNameNode.

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

nsabharwal — Wed, 10 Feb 2016 23:56:01 GMT

@Chris Nauroth How about when HA is not enabled?

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

cnauroth — Thu, 11 Feb 2016 02:16:32 GMT

@Neeraj Sabharwal, when HA is not enabled, HDP deployments would rely on the SecondaryNameNode for managing checkpoints.

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

nsabharwal — Thu, 11 Feb 2016 04:16:59 GMT

@Chris Nauroth right...So, the above comments are valid?

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

cnauroth — Thu, 11 Feb 2016 04:35:43 GMT

@Neeraj Sabharwal, it is not valid to link to documentation of the CheckpointNode or quote portions of it as a reference. The CheckpointNode and the SecondaryNameNode are similar, but different. The CheckpointNode is not run by HDP or any other distro I've encountered, so discussing it is highly likely to cause confusion. Instead, non-HA deployments would run the SecondaryNameNode.

Re: When runtime modifications are written to Edits log file in Name Node, is the Edits Log file getting updated on RAM or Local Disk?

nsabharwal — Thu, 11 Feb 2016 04:37:41 GMT

@Chris Nauroth Thanks for sharing your opinion on it. I do respect it 🙂

I was just curious to know more context behind your inputs. Appreciate it!!!