question Re: Why are the HDFS edit logs retained after a checkpoint operation is complete? in Archives of Support Questions (Read Only)

Why are the HDFS edit logs retained after a checkpoint operation is complete?

TerryP — Wed, 10 Feb 2016 23:50:19 GMT

In an HA environment, I see a fairly current fsimage file in the Namenode directory. But I also see edit files for the last 20+ days even though the checkpoint frequency is 6 hours/1M transactions.

Re: Why are the HDFS edit logs retained after a checkpoint operation is complete?

nsabharwal — Wed, 10 Feb 2016 23:55:06 GMT

@terry

See this http://hortonworks.com/blog/hdfs-metadata-directories-explained/

Re: Why are the HDFS edit logs retained after a checkpoint operation is complete?

TerryP — Thu, 11 Feb 2016 00:50:18 GMT

Pointed me to the right place to do more research.

Re: Why are the HDFS edit logs retained after a checkpoint operation is complete?

TerryP — Thu, 11 Feb 2016 00:53:07 GMT

The hdfs-site.xml property dfs.namenode.num.extra.edits.retained defaults to a value of 1000000 (1 million) Reducing this value will lower the number of edit logs retained after a checkpoint in complete.