Support Questions
Find answers, ask questions, and share your expertise

how namenode store metadata in RAM?

how namenode store metadata in RAM?

Explorer

I have confusion on the actual use of fsimage and edits log. can any explain on my below queries it will be really helpful for me.

Q1. why there are so many fsimage and edits log in namenode? what's the use of these many fsimage and edits log? if there is already schedule for checkpointing?

Q2 what's happening with old fsimage and edits log after checkpointing. when sn ( secondary NN) sends a new fsimage to namenode then whats the use of this new fsimage in name node. is namenode store the location of block as per the new fsimage (whatever the blocks name in new fsimage)

Q3 if namenode is running from last 4 years then it will store all the changes or metadata in RAM (or is there any logic to store old block location in some other place)? from the first of the year?

Q4 is namenode updating the metadata in RAM for every second? or is there any time peroid?

4 REPLIES 4
Highlighted

Re: how namenode store metadata in RAM?

Explorer

Please check my other answer for better understanding on this topic.

Q1. There is only one FSImage and one Edit log. Why do you suppose there are many of them?

Q2. When SN sends the new FSImage the old one will be overwritten.

"is namenode store the location of block as per the new fsimage"

I am not sure if I understand well but:

Of course it does. The new FSImages is created based on how the blocks are currently stored. NN does not need to adapt to FSImage.

Q3. No it does not.

FSimage is a snapshot of HDFS's. Shows the block locations, permissions, does not show historical data.

Edit log is a delta from last checkpoint. So it shows what changed since the last checkpoint.

The size of FSImage does not depend on time the NN running from. It depends on the size of HDFS namespace (number of files, users, etc).

Q4. You can set the snapshot period with

dfs.namenode.checkpoint.check.period

Another option is to limit the number of stored transactions. Use

dfs.namenode.checkpoint.txns
Highlighted

Re: how namenode store metadata in RAM?

Explorer

@abokor,

Q1- I have seen 2 fsimage always exist on SN. It stores two old FSimage for backup and rollback purpose but i don't know why it stores so many edits log. Once you will check the location of edits log you will get so many edits log.

I don't know what's the use of these many edits log.

Highlighted

Re: how namenode store metadata in RAM?

Explorer

They are the old edit logs. You can limit the number of retained files with dfs.namenode.num.checkpoints.retained setting.

Highlighted

Re: how namenode store metadata in RAM?

Explorer

@satya gaurav,

If you feel you got the answer for your question please accept one of the answers.