CDH 5.7.
This is intresting. The node that runs Cloudera Manager, Hue, HIve, Ozzie, reports a Clock Offset even with ntp running correctly. This node is not a Datanode or a Namenode. In this situation ALL services are reported as failed ("Bad") health except HDFS. I'm still able to browse the HDFS filesystem, run example MapReduce jobs and other services reported as "Bad" such as the Hue service, still work. Once I disable the Clock Offset ckeck on this node all serviced return to a good state within a minute. I can duplicate this behavor by enabling the check.
I'd like to see the actual output from the ntpq check, but fail to find anything in the logs that indicate errors.
- New installation.
- I have removed and reinstalled the NTP software
- Changed the ntp server pool.
- rebooted the troubled node.
- While ntp wasn't running, things were good.
All the host are in sync:
$ for i in `cat all_host `; do echo $i ; ssh $i date; done
node1
Mon Jul 11 14:01:44 CDT 2016
node2
Mon Jul 11 14:01:44 CDT 2016
node3
Mon Jul 11 14:01:44 CDT 2016
node4
Mon Jul 11 14:01:44 CDT 2016
node5
Mon Jul 11 14:01:44 CDT 2016
[root@node1~]# ntpq -c pe
remote refid st t when poll reach delay offset jitter
==============================================================================
+mirror 216.93.242.12 3 u 77 128 377 0.882 14.521 3.551
+cheezum.mattnor 129.7.1.66 2 u 6 128 377 23.015 13.455 2.413
*104.131.53.252 209.51.161.238 2 u 1 128 377 18.015 15.136 4.403
+srcf-ntp.stanfo 204.63.224.70 2 u 119 128 377 58.044 7.694 2.833
[root@node1~]# ntpdc -np
remote local st poll reach delay offset disp
=======================================================================
=208.53.158.34 10.104.101.224 3 128 377 0.00087 0.014521 0.07982
*104.131.53.252 10.104.101.224 2 128 377 0.01801 0.015136 0.07610
=67.18.187.111 10.104.101.224 2 128 377 0.02301 0.013455 0.08508
=171.66.97.126 10.104.101.224 2 128 377 0.05803 0.007694 0.09164
