One of hbase region server is stopping with below error:

subhash_parise3 — Thu, 26 May 2016 14:42:25 GMT

Please see the log here:

2016-05-26 18:25:18,671 INFO [regionserver/ambariagent14/10.76.64.14:16020] regionserver.HRegionServer: reportForDuty to master=ambariagent16,16000,1464267551115 with port=16020, startcode=1464267313347 2016-05-26 18:25:18,675 FATAL [regionserver/ambariagent14/10.76.64.14:16020] regionserver.HRegionServer: Master rejected startup because clock is out of sync org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server ambariagent14,16020,1464267313347 has been rejected; Reported time is too far out of sync with master. Time difference of 240124ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:348) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:325) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2272) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:894) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server ambariagent14,16020,1464267313347 has been rejected; Reported time is too far out of sync with master. Time difference of 240124ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:348) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1226) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2270) ... 2 more 2016-05-26 18:25:18,677 FATAL [regionserver/ambariagent14/10.76.64.14:16020] regionserver.HRegionServer: ABORTING region server ambariagent14,16020,1464267313347: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server ambariagent14,16020,1464267313347 has been rejected; Reported time is too far out of sync with master. Time difference of 240124ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:348) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745)

Re: One of hbase region server is stopping with below error:

rchintaguntla — Thu, 26 May 2016 14:52:25 GMT

There is time difference of more than half minute between master and region servers that's why getting clock is out of sync exception. You can set same time for master and regionservers machines or install ntp to maintain time sync among all the machines.

Re: One of hbase region server is stopping with below error:

subhash_parise3 — Thu, 26 May 2016 16:24:38 GMT

yes rajesh. i have changed the ranger server time into master time solves the issue. Thnx for your reply.

Re: One of hbase region server is stopping with below error:

subhash_parise3 — Thu, 26 May 2016 16:24:42 GMT

setup NTP service solves the issue.

Re: One of hbase region server is stopping with below error:

amit_singh1 — Thu, 06 Oct 2016 17:46:03 GMT

Login as the root user
Type the following command to install ntp
# yum install ntp
Turn on service
# chkconfig ntpd on

Re: One of hbase region server is stopping with below error:

Narendra_ — Wed, 17 Feb 2021 18:06:05 GMT

How to change set the timings of master and region server.Please provide us the steps.

Thanks,

Narendra

Re: One of hbase region server is stopping with below error:

VidyaSargur — Thu, 18 Feb 2021 04:52:12 GMT

Hi @Narendra_, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

question Re: One of hbase region server is stopping with below error: in Archives of Support Questions (Read Only)

One of hbase region server is stopping with below error:

Re: One of hbase region server is stopping with below error:

Re: One of hbase region server is stopping with below error:

Re: One of hbase region server is stopping with below error:

Re: One of hbase region server is stopping with below error:

Re: One of hbase region server is stopping with below error:

Re: One of hbase region server is stopping with below error: