Support Questions
Find answers, ask questions, and share your expertise

"java.io.IOException: error or interrupted while splitting logs" when starting HBase:

Highlighted

"java.io.IOException: error or interrupted while splitting logs" when starting HBase:

New Contributor

Hi, I'm using the Sandbox since 3 months without any problem. I tried to start HBase through Ambari today but it stopped instantly, so dug into the log and found that:

2016-08-08 12:16:03,536 FATAL [sandbox:16000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor]
2016-08-08 12:16:03,536 FATAL [sandbox:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
java.io.IOException: error or interrupted while splitting logs in [hdfs://sandbox.hortonworks.com:8020/apps/hbase/data/WALs/sandbox.hortonworks.com,16020,1470382618757-splitting] Task = installed = 1 done = 0 error = 1
at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:391)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:306)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitMetaLog(MasterFileSystem.java:297)
at org.apache.hadoop.hbase.master.HMaster.splitMetaLogBeforeAssignment(HMaster.java:1021)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:727)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:191)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1783)
at java.lang.Thread.run(Thread.java:745)
2016-08-08 12:16:03,536 INFO  [sandbox:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.

Any idea on how to fix this? I'm using the standard sandbox.

Thanks

Vincent

4 REPLIES 4
Highlighted

Re: "java.io.IOException: error or interrupted while splitting logs" when starting HBase:

Rising Star

@Vincent Peres sometimes if just powering off the VM without first taking down the services gracefully could cause the underlying filesystem to want to be checked before it can start up successfully. Kind of like a safety check. Sometimes this can be taken care of automatically but in some scenarios just running a manual check/fix is needed. Could you just try an hbase hbck command to see is any tables have an issue (see information here http://hbase.apache.org/0.94/book/hbck.in.depth.html)? I'd also look at the sandbox HBase Master UI to see if it shows any glaring issues http://sandbox.hortonworks.com:16010/master-status (or just hit the quick link through Ambari). If there are problem with a table the hbck command should help get you out of an issue.....hope this helps.

Re: "java.io.IOException: error or interrupted while splitting logs" when starting HBase:

Contributor

Hi Dan, just a small tip: when it comes to hbase documentation, we'd better refer to this latest one rather than hbase-0.94: http://hbase.apache.org/book.html#hbck.in.depth

Highlighted

Re: "java.io.IOException: error or interrupted while splitting logs" when starting HBase:

New Contributor

I actually deleted the file mentioned in the log hdfs://sandbox.hortonworks.com:8020/apps/hbase/data/WALs/sandbox.hortonworks.com,16020,1470382618757-splitting and then hbase started properly.

Highlighted

Re: "java.io.IOException: error or interrupted while splitting logs" when starting HBase:

Rising Star

NOTE the WAL is the write ahead log for all puts/deletes execute to a table. The WAL ensures Hbase has durable writes....did you happen to check to see if the file you purged had any content first? Since this is a sandbox may not matter so much for you.