Member since
12-21-2018
16
Posts
1
Kudos Received
0
Solutions
11-29-2018
03:27 PM
First, you should investigate in the log files to understand why this region became stuck in this PENDING_CLOSE state. Then, you can close and reassign this region. HBCK should be able to do this for you, or you can use the HBase shell commands to do it by hand.
... View more
08-03-2018
08:48 AM
Dump of Hbase Master.. 2018-08-03 12:59:58,968 ERROR [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-0] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.io.IOException: failed log splitting for chmcisprbddn01.chm.intra,16020,1503542011010, will retry
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:378)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:222)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn01.chm.intra,16020,1503542011010-splitting] Task = installed = 1 done = 0 error = 1
at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:290)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:429)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:402)
at org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:319)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:215)
... 4 more
2018-08-03 12:59:58,969 WARN [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-0] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is deprecated. Instead, use "hbase.client.scanner.timeout.period"
2018-08-03 12:59:58,971 INFO [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] handler.ServerShutdownHandler: Splitting logs for chmcisprbddn02.chm.intra,16020,1503542193852 before assignment; region count=0
2018-08-03 12:59:58,973 INFO [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] master.SplitLogManager: dead splitlog workers [chmcisprbddn02.chm.intra,16020,1503542193852]
2018-08-03 12:59:58,973 INFO [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] master.SplitLogManager: started splitting 1 logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn02.chm.intra,16020,1503542193852-splitting] for [chmcisprbddn02.chm.intra,16020,1503542193852]
2018-08-03 12:59:58,977 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn02.chm.intra%2C16020%2C1503542193852-splitting%2Fchmcisprbddn02.chm.intra%252C16020%252C1503542193852.default.1503542200899 acquired by chmcisprbddn07.chm.intra,16020,1533218036449
2018-08-03 12:59:59,005 INFO [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn02.chm.intra%2C16020%2C1503542193852-splitting%2Fchmcisprbddn02.chm.intra%252C16020%252C1503542193852.default.1503542200899 entered state: ERR chmcisprbddn07.chm.intra,16020,1533218036449
2018-08-03 12:59:59,006 WARN [main-EventThread] coordination.SplitLogManagerCoordination: Error splitting /hbase-unsecure/splitWAL/WALs%2Fchmcisprbddn02.chm.intra%2C16020%2C1503542193852-splitting%2Fchmcisprbddn02.chm.intra%252C16020%252C1503542193852.default.1503542200899
2018-08-03 12:59:59,006 WARN [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] master.SplitLogManager: error while splitting logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn02.chm.intra,16020,1503542193852-splitting] installed = 1 but only 0 done
2018-08-03 12:59:59,006 ERROR [MASTER_SERVER_OPERATIONS-CHMCISPRBDSN01:16000-2] executor.EventHandler: Caught throwable while processing event M_SERVER_SHUTDOWN
java.io.IOException: failed log splitting for chmcisprbddn02.chm.intra,16020,1503542193852, will retry
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:378)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:222)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error or interrupted while splitting logs in [hdfs://prodcluster/apps/hbase/data/WALs/chmcisprbddn02.chm.intra,16020,1503542193852-splitting] Task = installed = 1 done = 0 error = 1
... View more