Bad : The host's NTP service is not synchronized to any remote server. / Clock offset

[#1]

Hi,

since two days I get this message from CDH Maneger: Bad : The host's NTP service is not synchronized to any remote server.

all nodes are in sync with our centralized ntp infrastructure and the offset is smaller than one second:

[root@lintest01]:~ # pdsh -g cloudera ntpdate -u ntpserver1
hadoop01: 27 Oct 08:02:59 ntpdate[7126]: adjust time server xxx.xxx.xxx.xxx offset -0.020866 sec
hadoop13: 27 Oct 08:03:00 ntpdate[13383]: adjust time server xxx.xxx.xxx.xxx offset -0.031627 sec
hadoop18: 27 Oct 08:03:00 ntpdate[4948]: adjust time server xxx.xxx.xxx.xxx offset -0.034736 sec
hadoop15: 27 Oct 08:03:00 ntpdate[17460]: adjust time server xxx.xxx.xxx.xxx offset -0.023629 sec
hadoop17: 27 Oct 08:03:00 ntpdate[2297]: adjust time server xxx.xxx.xxx.xxx offset -0.028958 sec
hadoop14: 27 Oct 08:03:00 ntpdate[10184]: adjust time server xxx.xxx.xxx.xxx offset -0.016336 sec
hadoop19: 27 Oct 08:03:00 ntpdate[3260]: adjust time server xxx.xxx.xxx.xxx offset -0.029048 sec
hadoop16: 27 Oct 08:03:00 ntpdate[4632]: adjust time server xxx.xxx.xxx.xxx offset -0.002662 sec
hadoop20: 27 Oct 08:03:00 ntpdate[1047]: adjust time server xxx.xxx.xxx.xxx offset -0.013164 sec
hadoop12: 27 Oct 08:03:00 ntpdate[5675]: adjust time server xxx.xxx.xxx.xxx offset -0.010399 sec
hadoop02: 27 Oct 08:03:00 ntpdate[6904]: adjust time server xxx.xxx.xxx.xxx offset -0.023079 sec
hadoop04: 27 Oct 08:03:00 ntpdate[13129]: adjust time server xxx.xxx.xxx.xxx offset -0.004723 sec
hadoop09: 27 Oct 08:03:00 ntpdate[1449]: adjust time server xxx.xxx.xxx.xxx offset -0.007031 sec
hadoop07: 27 Oct 08:03:00 ntpdate[5187]: adjust time server xxx.xxx.xxx.xxx offset -0.009042 sec
hadoop10: 27 Oct 08:03:00 ntpdate[7991]: adjust time server xxx.xxx.xxx.xxx offset -0.005117 sec
hadoop08: 27 Oct 08:03:00 ntpdate[5223]: adjust time server xxx.xxx.xxx.xxx offset -0.008938 sec
hadoop05: 27 Oct 08:03:00 ntpdate[12730]: adjust time server xxx.xxx.xxx.xxx offset -0.016491 sec
hadoop06: 27 Oct 08:03:00 ntpdate[15570]: adjust time server xxx.xxx.xxx.xxx offset -0.010433 sec
hadoop11: 27 Oct 08:03:00 ntpdate[14559]: adjust time server xxx.xxx.xxx.xxx offset -0.019028 sec
hadoop03: 27 Oct 08:03:00 ntpdate[7603]: adjust time server xxx.xxx.xxx.xxx offset -0.024363 sec

tha ntpd is running on all nodes.

any ideas whats going wrong?

Best Regards

Oliver

[#2]

the documentation says: To troubleshoot, run 'ntpq -np' or 'chronyc sources' on the host. The output should contain a row for each of the host's NTP servers. The row starting with a '*' (if ntpdc) or '^*' (if chronyc) contains the peer to which the host is currently synchronized. If no row starts with a '*' or '^*', this indicates that the host is not currently synchronized. Make sure you have a pool of valid ntp servers in your ntp.conf (if ntpdc) or chrony.conf (if chronyc).

tha output on my server looks like that:

hadoop02:      remote           local      st poll reach  delay   offset    disp
hadoop02: =======================================================================
hadoop02: =xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx      2   64  377 0.00148 -0.045714 0.12851
hadoop02: =xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx      2   64  377 0.00131 -0.041083 0.12852
hadoop02: =xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx      2   64  377 0.00177 -0.037501 0.12849

no line starts with *, only =

hu? any ideas? 🙂

[#3]

1.  If you did not install ntp then install ntp using following command

# yum install ntp

2.To troubleshoot, run

$ ntpq -np

In my case I got following

[root@quickstart cloudera]# ntpq -np

remote        refid   st t when poll reach   delay offset jitter

==============================================================================

123.108.200.124 80.87.128.222 4 u 2   64 1   27.217  6881.89  0.686

If the row starting with a '*' (if ntpdc) or '^*' (if chronyc) contains the peer to which the host is currently synchronized. If no row starts with a '*' or '^*', this indicates that the host is not currently synchronized.

3.To achieve this edit the /etc/ntp.conf

Add following lines into /etc/ntp.conf file

restrict <ipaddress> netmask 255.255.255.0 nomodify ntrap

logfile /var/log/ntp.log

4.Save the file and run ntpq -np 

I got "*" before the line

[root@quickstart cloudera]# ntpq -np

remote refid   st t when poll reach   delay offset jitter

==============================================================================

*123.108.200.124 80.87.128.222 4 u   39 64  377 23.909  -24.388 22.724

The * symbol before the  line will appear.

Now check Host’s health you will find:-

ntp  is set on cloudera user interface.

[#4]

solomonchinni has two typographical errors in their response:

ntrap should be notrap

netmask should be mask

The resulting corrected line should be:

restrict <ipaddress> mask 255.255.255.0 nomodify notrap

The <ipaddress> should be the start of the IP address range that you want to be defined by the provided mask.

For example, if your IP address range in 192.168.1.1 through 192.168.1.255, you'll want:

restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

[#5]

Hi,

     There are two ip addresses after running ntpq -np, remote and refid, do you mean we should use remote ip?

     Thank you for sharing your knowledge!