Actual use of FSimage and Edits log?

[#1]

I have a confusion on the actual use of fsimage. can any explain on my below queries? it will be really helpful for me.

In so many articles it is written there fsimage and edits log use only at the time of namenode restart, but my question is suppose the namenode was running from last 10 years then namonde store all the changes (Including block location) in metadata happened from last 10 years in RAM only? it will not a size or performance issue for namenode? or namenode store the metadata as per new fsimage. how namenode using the new fsimage which is received by SN.?

[#2]

@satya gaurav

Welcome at Hortonworks Community.

I feel some misunderstanding here.

• FSimage is a point-in-time snapshot of HDFS's namespace.
• Edit log records every changes from the last snapshot. The last snapshot is actually stored in FSImage.

So edit log records the changes that was taken on the file system and (to avoid the problem you came up with) in a certain period (based on your cluster settings) Standby Node or SN will merge edit log to FSImage and returns with a new FSImage.

That means HDFS does not need to store all the changes from the last 10 years. It stores the changes only from the last checkpoint.

The size of FSImage does not depend on time the NN running from. It depends on the size of HDFS namespace (number of files, users, etc).

You can set the snapshot period with

dfs.namenode.checkpoint.check.period

Another option is to limit the number of stored transactions. Use

dfs.namenode.checkpoint.txns

Hope it helps. If it still not clear enough feel free to ask.

[#3]

Hi Abokor,

Thanks a lot for your reply, but still I have one query as per your explanation edits log and fsimage store changes only from the last checkpoint.So if any user wants to get the file information or any data which is 10 years old then how it will get the metadata information for that particular file or data? It will access the metadata (block location,etc) only from RAM or it will use some other process?

[#4]

"as per your explanation edits log and fsimage store changes only from the last checkpoint"

No.

Your HDFS state is stored in FSImage. All the changes (rename, delete, permission change) to last checkpoint will be stored in edit log. To avoid that the edit log to be too large sometimes the SN applies the changes from edit log in FSImage. That means FSImage stores your whole HDFS state not only from the last checkpoint.

With other words, the checkpoint is actually the FSImage.

This whole process is needed because it is hard to write FSImage it is easier to write the changes into edit log, but edit log could be too large so merge is needed in certain period.

Example:

You have a file a.txt and you rename to b.txt. After you rename b.txt to c.txt

FSImage shows you have a.txt and in edit log it stores you had two renames (a -> b, b -> c)

After SN merges FSImage and edit log:

Edit log will be empty and FSImage will show you have c.txt. There will be no any record for a.txt.

Does it answer you question?

[#5]

@abokor,

Hi,

Thanks man!

My question was suppose there is a file with the name of A.txt and it has created before 10 years and there is no change from last 10 years in this file. Now if any client wants to access the same file. so from where it will get the metadata (block location, permission, name,etc) from RAM only(since NN store all metadata in RAM)? or somewhere else ?

[#6]

Yes, it will use RAM since it is the fastest way to reach the required data. That is why Apache recommends to use lot of memory on NN.

[#7]

@satya gaurav,

If you feel you got the answer for your question please accept one of the answers.

[#8]

Hi @abokor,

Let's say, I've a.txt in fsimage and renamed it to b.txt. Before this change is merged with fsimage, if I query the a.txt, will I get the file as fsimage still have it ? If not how it's dealt with ?

[#9]

@Ranjit masetty ,

No, you won't. Even if you have a record for a.txt in FSImage it will check change log to ensure it was not deleted or renamed. Since edit log shows you renamed a.txt NameNode will return with no file.

Parsing edit log is a very fast process.

[#10]

@Ranjit masetty Though I'm too late to answer your query but I have gone through all the previous comments, and I have tested your query on my cluster. And the answer to your question is NO. You won't be able to access a.txt once you have changed the file name to b.txt because hdfs changes are really quick and the changes will be stored in edit_logs. So, you won't be able to access your file as a.txt.