hbase region servers failing in the cluster [ Edited ]

[cdhhadoop]

Hello All,

Hbase region servers are in bad health in our cluster. I restarted the region servers, then I tried restarting hbase service and then entire cluster restart too.

But didn't help. There were errors related to clock out of sync, so synched time.but no luck. then there was address already in use. So stopped hbase service and restarted, but all ports were open.

After restarting the cluster, I see below error in the logs.

Caught M_META_SERVER_SHUTDOWN, count=1
java.io.IOException: failed log splitting for servervnanme,60020,1536743351491, will retry
at org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:84)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://nameservice-ha/hbase/WALs/servername,60020,1536743351491-splitting]
Task = installed = 1 done = 0 error = 0
at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:289)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:305)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:296)

Can anyone please help me to resolve the issue?

Thanks,

Priya

[TomasTF]

You should use ntp or chrony to synchronize clocks. If they are used, and the clocks are out of sync, maybe some issue is on the network. Regarding the Hbase restart, I would do a Stop -> then check on all nodes that no hbase is running and then start. 

[cdhhadoop]

@TomasTF,

Thanks for inputs. I synchronized clocks but no luck. And then have already tried hbase restart, but it didn't help.

Please suggest.

Thanks,
Priya