Support Questions
Find answers, ask questions, and share your expertise

One of hbase region server is stopping with below error:

Solved Go to solution
Highlighted

One of hbase region server is stopping with below error:

Super Collaborator

Please see the log here:

2016-05-26 18:25:18,671 INFO [regionserver/ambariagent14/10.76.64.14:16020] regionserver.HRegionServer: reportForDuty to master=ambariagent16,16000,1464267551115 with port=16020, startcode=1464267313347 2016-05-26 18:25:18,675 FATAL [regionserver/ambariagent14/10.76.64.14:16020] regionserver.HRegionServer: Master rejected startup because clock is out of sync org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server ambariagent14,16020,1464267313347 has been rejected; Reported time is too far out of sync with master. Time difference of 240124ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:348) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:325) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2272) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:894) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server ambariagent14,16020,1464267313347 has been rejected; Reported time is too far out of sync with master. Time difference of 240124ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:348) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1226) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2270) ... 2 more 2016-05-26 18:25:18,677 FATAL [regionserver/ambariagent14/10.76.64.14:16020] regionserver.HRegionServer: ABORTING region server ambariagent14,16020,1464267313347: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server ambariagent14,16020,1464267313347 has been rejected; Reported time is too far out of sync with master. Time difference of 240124ms > max allowed of 30000ms at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:388) at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:262) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:348) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745)

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: One of hbase region server is stopping with below error:

There is time difference of more than half minute between master and region servers that's why getting clock is out of sync exception. You can set same time for master and regionservers machines or install ntp to maintain time sync among all the machines.

View solution in original post

6 REPLIES 6
Highlighted

Re: One of hbase region server is stopping with below error:

There is time difference of more than half minute between master and region servers that's why getting clock is out of sync exception. You can set same time for master and regionservers machines or install ntp to maintain time sync among all the machines.

View solution in original post

Highlighted

Re: One of hbase region server is stopping with below error:

Super Collaborator

yes rajesh. i have changed the ranger server time into master time solves the issue. Thnx for your reply.

Highlighted

Re: One of hbase region server is stopping with below error:

New Contributor

How to change set the timings of master and region server.Please provide us the steps.

 

Thanks,

Narendra

Re: One of hbase region server is stopping with below error:

Community Manager

Hi @Narendra_, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.


Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Highlighted

Re: One of hbase region server is stopping with below error:

Super Collaborator

setup NTP service solves the issue.

Highlighted

Re: One of hbase region server is stopping with below error:

New Contributor
  1. Login as the root user

    Type the following command to install ntp

    # yum install ntp

  2. Turn on service

    # chkconfig ntpd on