Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

One of hbase region server is stopping with below error:

avatar
Super Collaborator

Please see the log here:

2016-05-26 18:25:18,671 INFO [regionserver/ambariagent14/10.76.64.14:16020] regionserver.HRegionServer: reportForDuty to master=ambariagent16,16000,1464267551115 with port=16020, startcode=1464267313347 2016-05-26 18:25:18,675 FATAL [regionserver/ambariagent14/10.76.64.14:16020] regionserver.HRegionServer: Master rejected startup because clock is out of sync org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server ambariagent14,16020,1464267313347 has been rejected; Reported time is too far out of sync with master. Time difference of 240124ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:348) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:325) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2272) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:894) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server ambariagent14,16020,1464267313347 has been rejected; Reported time is too far out of sync with master. Time difference of 240124ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:348) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1226) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2270) ... 2 more 2016-05-26 18:25:18,677 FATAL [regionserver/ambariagent14/10.76.64.14:16020] regionserver.HRegionServer: ABORTING region server ambariagent14,16020,1464267313347: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server ambariagent14,16020,1464267313347 has been rejected; Reported time is too far out of sync with master. Time difference of 240124ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:348) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745)

1 ACCEPTED SOLUTION

avatar

There is time difference of more than half minute between master and region servers that's why getting clock is out of sync exception. You can set same time for master and regionservers machines or install ntp to maintain time sync among all the machines.

View solution in original post

6 REPLIES 6

avatar

There is time difference of more than half minute between master and region servers that's why getting clock is out of sync exception. You can set same time for master and regionservers machines or install ntp to maintain time sync among all the machines.

avatar
Super Collaborator

yes rajesh. i have changed the ranger server time into master time solves the issue. Thnx for your reply.

avatar
Explorer

How to change set the timings of master and region server.Please provide us the steps.

 

Thanks,

Narendra

avatar
Community Manager

Hi @Narendra_, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Super Collaborator

setup NTP service solves the issue.

avatar
New Contributor
  1. Login as the root user

    Type the following command to install ntp

    # yum install ntp

  2. Turn on service

    # chkconfig ntpd on