question Token expiry issue due to System time on machines. in Archives of Support Questions (Read Only)

Token expiry issue due to System time on machines.

pradeep_tp — Thu, 04 Feb 2016 23:11:08 GMT

I am struggling to fix the issue that I am facing while executing hadoop mareduce jobs in my cluster. I am running the mapreduce job on the cluster created through Ambari (not sandbox). The cluster has 4 nodes (including the master node). Following is the error that I get

This token is expired. current time is 1454617494914 found 1454598336617 Note: System times on machines may be out of sync. Check system time and time zones.

I checked the time on all the nodes. I found that, except the master node, time on all the other nodes were incorrect. So I manually corrected (ntpd was failing to connect to servers) the time on all the nodes.

Searching the internet, I found that there is a setting 'yarn.resourcemanager.rm.container-allocation.expiry-interval-ms' which can be used to increase the lifespan of the container. I could not find this setting anywhere in the advanced configuration on the Ambari dashboard. Can anyone help me understand what is going on ?

Re: Token expiry issue due to System time on machines.

aervits — Thu, 04 Feb 2016 23:11:09 GMT

install NTP @Pradeep kumar

Re: Token expiry issue due to System time on machines.

aervits — Thu, 04 Feb 2016 23:12:12 GMT

@Pradeep kumar

# Setup NTPD
chkconfig --list ntpd
chkconfig ntpd on
service ntpd stop
ntpdate pool.ntp.orgservice 
ntpd start

Re: Token expiry issue due to System time on machines.

aervits — Thu, 04 Feb 2016 23:13:20 GMT

@Pradeep kumar I also think the issue is not with YARN but with kerberos.

Re: Token expiry issue due to System time on machines.

nsabharwal — Thu, 04 Feb 2016 23:14:08 GMT

@Pradeep kumar

This is the exact root cause

Do you know why ntpd is failing?

Re: Token expiry issue due to System time on machines.

aervits — Thu, 04 Feb 2016 23:15:50 GMT

above is for RHEL6 for RHEL7 below

# Setup NTPD
yum install -y ntp
systemctl is-enabled ntpd
systemctl enable ntpd


# enable firewall rules for ntp
firewall-cmd --add-service=ntp --permanent
firewall-cmd --reload
systemctl stop ntpd
ntpdate pool.ntp.org
systemctl start ntpd
systemctl status ntpd
echo "wait 30 sec for time to synchronize"
sleep 30
ntpq -p
date -R

Re: Token expiry issue due to System time on machines.

pradeep_tp — Fri, 05 Feb 2016 00:03:23 GMT

Thanks Neeraj. I wish I could update the time using the ntpd, but I tried all commands to update the system time, but I kept getting the error "4 Feb 21:30:55 ntpdate[12169]: no server suitable for synchronization found". I have gone through a lot of materials on internet that discusses about this error, but none of the suggestions helped me, so I thought of doing it manually. Okay. I will check with my company network support and see if it is a problem with firewall, due to which ntpd is not able to sycn with the server.

Re: Token expiry issue due to System time on machines.

pradeep_tp — Fri, 05 Feb 2016 00:05:39 GMT

I have installed the cluster using CentOS. So, it would be great if you could post the CentOS version for setting the firewall rule. This could be reason, why my nodes are not able to contact the time servers. Many thanks.

Re: Token expiry issue due to System time on machines.

aervits — Fri, 05 Feb 2016 00:10:17 GMT

here you go @Pradeep kumar link

Re: Token expiry issue due to System time on machines.

pradeep_tp — Fri, 05 Feb 2016 00:31:37 GMT

I have tried disabling the firewall and running the command '/usr/sbin/ntpdate pool.ntp.org'. But, I am getting the error "no server suitable for synchronization found".

Re: Token expiry issue due to System time on machines.

pradeep_tp — Fri, 05 Feb 2016 00:37:24 GMT

Also, I would like to understand, why setting the time manually will not resolve the problem of time synching. If I type "date", it shows me almost the same time on all the nodes now. The dates are same, but the time varies only by a few seconds.

Re: Token expiry issue due to System time on machines.

aervits — Fri, 05 Feb 2016 01:05:30 GMT

@Pradeep kumar its a common error, Google search gave me this Link

Re: Token expiry issue due to System time on machines.

aervits — Fri, 05 Feb 2016 01:08:33 GMT

@Pradeep kumar also make sure your firewall accepts all traffic from servers in the cluster. You can open ports granularly or allow all traffic from node. Refer to Centos docs for instructions

Re: Token expiry issue due to System time on machines.

nsabharwal — Fri, 05 Feb 2016 01:13:03 GMT

@Pradeep kumar Please see this http://stackoverflow.com/questions/21432002/ntp-configuration-without-internet

Re: Token expiry issue due to System time on machines.

pradeep_tp — Fri, 05 Feb 2016 21:03:38 GMT

Thanks Neeraj for your support. The problem was NTPD. The problem occurred because my nodes could not reach the known ntpd time servers. So I got the address of an internal ntpd server in my company and everything started working fine.

Re: Token expiry issue due to System time on machines.

pradeep_tp — Fri, 05 Feb 2016 21:04:48 GMT

Thanks Artem for your support. The problem was not with the firewall, but the nodes were not able to reach the known ntpd time servers.

Re: Token expiry issue due to System time on machines.

aervits — Fri, 05 Feb 2016 21:40:04 GMT

@Pradeep kumar great, glad to help.

Re: Token expiry issue due to System time on machines.

stevel — Sun, 07 Feb 2016 02:30:37 GMT

a few seconds isn't going to matter. Kerberos and the security system is fussy about clocks.

you can usually set your network switch up as an NTP server, so they can all sync with that. Or turn one of your machines into the NTP server and again, make it a reference source of time. Ideally, if detached from the network, you could hook up a GPS unit and run gpsd to be as accurate as pretty much everything else on the internet