Reply
Contributor
Posts: 25
Registered: ‎06-13-2017

Re: kudu service are getting down frequently

You need to configure NTP correctly. "Four NTP servers is the recommended minimum. Four servers protects against one incorrect timesource, or "falseticker". " for tips in configuring NTP. https://access.redhat.com/solutions/58025
Expert Contributor
Posts: 334
Registered: ‎01-25-2017

Re: kudu service are getting down frequently

We have NTP servers, wondering why only Kudu impacted.

NTP and clock offset impacted a complete node and not a service.
New Contributor
Posts: 2
Registered: ‎10-25-2016

Re: kudu service are getting down frequently

Thank you very much.But I still have a trouble,there is no edit blank for me to change,and only a check box.How should I add the parameter to it..... 抑制参数验证:Kudu Service Environment Advanced Configuration Snippet (Safety Valve) Kudu(服务范围)   抑制参数验证:Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile Kudu(服务范围)   抑制参数验证:Kudu Service Advanced Configuration Snippet (Safety Valve) for kudu-monitoring.properties Kudu(服务范围)   抑制参数验证:Service Monitor Derived Configs Advanced Configuration Snippet (Safety Valve) Kudu(服务范围)  
Expert Contributor
Posts: 113
Registered: ‎02-15-2016

Re: kudu service are getting down frequently

you need to be login as an administrator to add these values.
Highlighted
Expert Contributor
Posts: 113
Registered: ‎02-15-2016

Re: kudu service are getting down frequently

I was able to resolve it .

 

if you see error with ntptime mostly kudu service will go down , so you have to restart ntpd and then this error will go .

[root@wcw0hd3dn02 ~]# ntptime

ntp_gettime() returns code 5 (ERROR)

  time dce7466c.fc37b000  Sun, Jun 11 2017  0:32:44.985, (.985225),

  maximum error 16000000 us, estimated error 16 us, TAI offset 0

ntp_adjtime() returns code 5 (ERROR)

  modes 0x0 (),

  offset 0.000 us, frequency 0.000 ppm, interval 1 s,

  maximum error 16000000 us, estimated error 16 us,

  status 0x4041 (PLL,UNSYNC,MODE),

  time constant 7, precision 1.000 us, tolerance 500 ppm,

 

 

this error comes if you run ntp with -x option 

 

[root@wuwcw0hd3mn01 ~]# ps -ef|grep ntp
root 3183 2731 0 10:38 pts/0 00:00:00 grep ntp
ntp 20736 1 0 Jun13 ? 00:00:19 ntpd -x -u ntp:ntp -p /var/run/ntpd.pid -g

 

remove -X from belwo file and restart ntp 

[root@wuwcw0hd3mn01 ~]# more /etc/sysconfig/ntpd
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid -g"

 

wait for ntp to synchronize ,after that i didnt see any issue in kudu service so far.

 

ntp_gettime() returns code 0 (OK)
time dcf260c3.66c6abfc Mon, Jun 19 2017 10:40:03.401, (.401469911),
maximum error 394157 us, estimated error 345 us, TAI offset 0
ntp_adjtime() returns code 0 (OK)
modes 0x0 (),
offset -707.277 us, frequency 20.094 ppm, interval 1 s,
maximum error 394157 us, estimated error 345 us,
status 0x6001 (PLL,NANO,MODE),
time constant 10, precision 0.001 us, tolerance 500 ppm,

 

 

check this https://access.redhat.com/solutions/38542

 

 

Cloudera Employee
Posts: 64
Registered: ‎04-08-2014

Re: kudu service are getting down frequently

Thank you @MSharma for updating us on your findings!

 

That is very interesting. It sounds like you enabled NTP stepping (which -x disables) whereas before it could only use slewing. Apparently stepping has kept you from falling too far out of sync from the time source for Kudu to tolerate.

 

I just checked and on one of the more long-lived and stable test environments I periodically use (note: it is NOT a production system) where I have run many different versions of Kudu over the years I do not have -x set in OPTIONS. On that machine (running CentOS 6.6) there is only the following in /etc/sysconfig/ntpd:

 

# Drop root to id 'ntp:ntp' by default.
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g"

 

It may be worth noting that this machine also has the following set in /etc/ntp.conf:

 

# tinker panic 0 instructs NTP not to give up if it sees a large jump in time.
# This is important for coping with large time drifts and also resuming virtual
# machines from their suspended state.
tinker panic 0 # Permit time synchronization with our time source, but do not # permit the source to query or modify the service on this system. restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery # Drift file. Put this in a directory which the daemon can write to. # No symbolic links allowed, either, since the daemon updates the file # by creating a temporary in the same directory and then rename()'ing # it to the file. driftfile /var/lib/ntp/drift

I can't tell you whether or not this is an ideal NTP configuration, or whether it is fully correct, but it seems stable.

 

For those who want more insight into what -x and slewing means, I'd recommend looking at the ntpd(8) man page and doing a Find for the keyword "slew": https://linux.die.net/man/8/ntpd

 

For those having problems with NTP stability in general, also consider reading through the "NTP Debugging Techniques" section of the Official NTP Documentation: http://doc.ntp.org/4.2.6p5/debug.html

New Contributor
Posts: 3
Registered: ‎06-29-2017

Re: kudu service are getting down frequently

I have been fighting the same problem. MSharma's instructions worked for me as well. Still not really clear on what it does (will read the slewing link provided by mpercy), but it at least works.
New Contributor
Posts: 3
Registered: ‎06-29-2017

Re: kudu service are getting down frequently

Just a quick follow-up on my note, above.

It seems to me that Kudu has an incompatibility with NTP slew. I have spent a solid day testing various scenarios, and if slew is on, Kudu won't start (or will eventually crash).

This does not seem to be related to time, as the time does not appear to creep away from what is expected.

Just that little "-x" on the ntpd command (either on the command line, or in the /etc/sysconfig/ntpd file) makes the different. With it, Kudu won't start, or will crash. Without it, all is fine.

I am using Kudu (parcel) with CDH 5.10.1.

For now, I will just continue without slew, and I post this in the hopes of helping others as well as follow on comments of other solutions. Certainly, I'd be interested in any one with a working Kudu to try to turn on slew and restart ntpd. Then does Kudu start correctly?

Thanks!!!!
Cloudera Employee J-D
Cloudera Employee
Posts: 60
Registered: ‎07-08-2013

Re: kudu service are getting down frequently

Any chance some of you are running on Azure? It has known issues with ntp: https://social.msdn.microsoft.com/Forums/azure/en-US/8c0a1026-0b02-405a-848e-628e68229eaf/i-have-a-l...

New Contributor
Posts: 3
Registered: ‎06-29-2017

Re: kudu service are getting down frequently

Not running Azure.
Announcements