Clock Offset Bad on cloudera manager server

[#1]

Hi All,

I have a test Cloudera Cluster composed by three servers(one with the cloudera manager and 2 data nodes). Sometime, but no more than twice a day, it happens that arrives an alert about Clock offset bad for a node. Once I log in into the server of the alert I have checked the service ntp with the command service ntp status and it seems to be  ok giving me this output telling me is running: 

Oct 30 12:46:25 mbesrvcdrmw1.mbeitaly.mbe.lan ntpd[5586]: 0.0.0.0 0615 05 clock_sync
Oct 30 12:46:26 mbesrvcdrmw1.mbeitaly.mbe.lan ntpd[5586]: 0.0.0.0 c618 08 no_sys_peer

I have tried to execute this command ntpq -np on all three servers and, for every one, there's in the output " * " in the beginning of a server line demonstrating the sync it's active.

I have also tried to restart the ntp service with service ntpd  restart and it seems to work but sometime the error reappears but spontaneously, after 15-20 minutes, fix itself automatically.

You can find in the attach the full output of these two commands. 

Do you have any ideas about how to fix this matter permanently ?

Many thanks in advance

Stefano 

[#2]

Hi @teolux ,

Do you have chronyd running on this host? Cloudera Manager agent will run the following command to determine if chronyd is running:

# pidof chronyd

If there is a chronyd process running, then the above command will return the PID of that process. Please run the "pidof chronyd" command and see what the output is. If there is a PID returned, use "ps aux | grep <pid>" to find out what that process actually is.

If you do not want to use chronyd, then disable that. Also restart Cloudera manager agent afterwards.

Thanks and hope this helps,
Li

[#3]

Hello  @lwang ,

Many thanks for your answer. I've just performed the command:

# pidof chronyd

on my machine and I have not installed cronyd  on the machines due to no PID returned. If I understand well this is like a tool used to sync the clock of the Operative system to the NTP server clock. Do you think it could be a good thing to install it for a permanent fix of the ntp clock issue ? Or maybe the restart of the ntpd service?

Today I have still received the alert but after 20-30 minutes spontaneously everything returned green then it's not a serious block problem. 

Many thanks in advance

I'm available for any clarification

Regards,

Teolux

Teolux 

[#4]

Hi @teolux ,

You may want to read this thread:

https://community.cloudera.com/t5/Support-Questions/Should-Cloudera-NTP-use-Chrony-or-NTPD/td-p/5598...

Basically Either NTP or Chrony can be used.

Thanks,
Li

[#5]

Hi @teolux ,

I am following up to see if you will be installing chronyd? Hope you have resolved the issue already.

Thanks,

Li