Understanding HDFS Checkpoint Triggers: dfs.namenode.checkpoint.period vs dfs.namenode.checkpoint.txns

Question

Checkpointing process is defined in 2 configuration parameters:

dfs.namenode.checkpoint.period -> 10 hr which will set the time for every checkpoint.

Aslos dfs.namenode.checkpoint.txns is by default set to 1 million. What does this configuration do, in apache wiki website it says this configurations sets the number of uncheckpointed transactions.

In the first configuration we are setting it for every 10hrs, in such scenario what is the necessity for setting value for uncheckpointed transactions.

dineshc · Accepted Answer

@Viswa dfs.namenode.checkpoint.period – The number of seconds between two periodic checkpoints.   dfs.namenode.checkpoint.txns – The standby will create a checkpoint of the namespace every ‘dfs.namenode.checkpoint.txns’ transactions, regardless of whether ‘dfs.namenode.checkpoint.period’ has expired. A real life analogy is how we take our car for regular maintenance service. dfs.namenode.checkpoint.period >> number of months (say 6 months) dfs.namenode.checkpoint.txns >> number of miles driven (say 5000 miles)   You will have to take your car for service if: It has been 6 months since your car’s last service and you may have driven less than 5000 miles OR It has not been 6 month since your car’s most recent service but you have driven 5000 miles already.   Thus a new checkpoint will be created if either the checkpoint period is reached or number of unchecked transactions has maxed out, whichever happens first.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.