question Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions in Support Questions

NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

suresh_b_k — Tue, 31 May 2016 18:47:13 GMT

HI Team

The 'NameNode Last Checkpoint' alert description says "This service-level alert will trigger if the last time that the NameNode performed a checkpoint was too long ago. It will also trigger if the number of uncommitted transactions is beyond a certain threshold."

I got alert on HA, Saying checkpoint happened too long ago.. How to solve this issue.

HDP : 2.3

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

suresh_b_k — Tue, 31 May 2016 19:43:14 GMT

Do i need to run any commands? Manually to update the checkpoint .

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

suresh_b_k — Tue, 31 May 2016 22:16:54 GMT

Can some one help ????

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

divakarreddy_a — Tue, 31 May 2016 22:31:00 GMT

@suresh krish

Which version of Ambari are you using? it seems your issue is fixed in Ambari 2.4

details are here:

https://issues.apache.org/jira/browse/AMBARI-15953

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

suresh_b_k — Tue, 31 May 2016 23:35:41 GMT

Ambari 2.1

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

suresh_b_k — Tue, 31 May 2016 23:36:35 GMT

What is the actual cause. Is it really not checkpointing the edits or its just a ignorable alert

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

bandarusridhar1 — Tue, 31 May 2016 23:49:42 GMT

Restarting the Namenode will fix your issue, Before restarting run below commands

hdfs dfsadmin -safemode enter
hdfs dfsadmin -saveNamespace
hdfs dfsadmin -safemode leave

Now restart NN, JN.

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

divakarreddy_a — Wed, 01 Jun 2016 00:08:59 GMT

I'm not sure if we can ignore this alerts but it seems issue with Ambari Alert script and they fixed it in 2.4. I would recommend to check with HDP while creating support ticket to get confirmation.

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

vpoornalingam — Wed, 01 Jun 2016 18:58:04 GMT

@suresh krish

Is this a test system? Are the values for the following modified from default

dfs.namenode.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints
dfs.namenode.checkpoint.txns, set to 1 million by default, defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not been reached.

You could go to the Namenode current folder and check when was the last fsimage created. Was the cluster down for long? You may want to review the above two parameters and check the timestamps in the current folder and find why automatic checkpointing is not happening.

You could use the steps provided by @Sri Bandaru to do a checkpoint manually if you could get downtime.

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

Yukti — Mon, 14 Nov 2016 17:47:51 GMT

@vpoornalingam I have checked the above two values: value of dfs.namenode.checkpoint.period is set to 6 hours. Does this creates the above mentioned alerts?

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

Yukti — Mon, 14 Nov 2016 17:50:19 GMT

@SBandaru

I have performed the above steps but after sometime these alerts are coming again and again. So can you please suggest something to fix this issue permanentaly.

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

nuspengcheng — Sat, 07 Jan 2017 12:53:44 GMT

@Yukti Agrawal , I have met just the same problem, after performing the above steps ,then after sometime these alerts are coming again and again , did you solve this problem ? I set dfs.namenode.checkpoint.period to 1 hour , but it seems it did not work, since I checked the fsimage file which is not generated automatically per hour , thanks

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

elkan1788 — Thu, 09 Mar 2017 15:46:30 GMT

It's seems not really resolve problem. It's still show up next time. What's the final way?

Re: NameNode Last Checkpoint script alert definition does not trigger based on uncommitted transactions

dashielb — Wed, 17 Apr 2019 22:23:31 GMT

Hello,

Restarting the Standby NameNode did not resolve the issue. Then when trying to restart the Active NameNode Ambari told me to SSH into my Active NameNode server and:

Enter Safe Mode

sudo su hdfs -l -c 'hdfs dfsadmin -safemode enter'

Safe mode is ON in xx.xxx.xxx.xxx:8020

The save Namespace

root@xxxxx:~# sudo su hdfs -l -c 'hdfs dfsadmin -saveNamespace'

Save namespace successful for xx.xxx.xxx.xxx:8020

After that restarting both NameNodes cleared the Alerts.

Regards.