Member since
04-11-2018
10
Posts
0
Kudos Received
0
Solutions
06-24-2018
02:16 PM
I did follow the same step to check for the corrupted blocks which are causing the HBase Master start up error and then deleted the same. Once done restarted the HBase services successfully.
... View more
06-23-2018
07:29 AM
Yes...HDFS has reported missing blocks...NameNode Critical error : Total Blocks:[99], Missing Blocks:[7]
... View more
06-21-2018
07:54 AM
I've set up my HCP cluster with 4 nodes. But I observed that my HBase Master never starts up after cluster restart. If I leave the cluster up and running from the installation time, then HBase works fine. But once I shut down the cluster and the try to start again Master fails to start. When I looked into the master log I found out that, during shut down time there was the following warning entries: ------------------------------------------------------------------------------------------ 2018-06-14 14:48:14,099 WARN [MASTER_META_SERVER_OPERATIONS-vmbdsiwbdn2:16000-3] master.AssignmentManager: Unable to determine a plan to assign {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
2018-06-14 14:48:15,099 WARN [MASTER_META_SERVER_OPERATIONS-vmbdsiwbdn2:16000-3] master.AssignmentManager: Can't move 1588230740, there is no destination server available.
2018-06-14 14:48:15,099 WARN [MASTER_META_SERVER_OPERATIONS-vmbdsiwbdn2:16000-3] master.AssignmentManager: Unable to determine a plan to assign {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
2018-06-14 14:48:16,099 WARN [MASTER_META_SERVER_OPERATIONS-vmbdsiwbdn2:16000-3] master.AssignmentManager: Can't move 1588230740, there is no destination server available.
2018-06-14 14:48:16,099 WARN [MASTER_META_SERVER_OPERATIONS-vmbdsiwbdn2:16000-3] master.AssignmentManager: Unable to determine a plan to assign {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}
2018-06-14 14:48:17,099 WARN [MASTER_META_SERVER_OPERATIONS-vmbdsiwbdn2:16000-3] master.AssignmentManager: Can't move 1588230740, there is no destination server available. ---------------------------------------------------------------------------------------------- And on Cluster restart, I found the following warning entries in the master log file: ---------------------------------------------------------------------------------------------- 2018-06-15 14:12:41,118 INFO [vmbdsiwbdn2:16000.activeMasterManager] hdfs.DFSClient: No node available for BP-1307428289-10.0.0.4-1528240888625:blk_1073741830_1006 file=/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2018-06-15 14:12:41,118 INFO [vmbdsiwbdn2:16000.activeMasterManager] hdfs.DFSClient: Could not obtain BP-1307428289-10.0.0.4-1528240888625:blk_1073741830_1006 from any node: java.io.IOException: No live nodes contain block BP-1307428289-10.0.0.4-1528240888625:blk_1073741830_1006 after checking nodes = [], ignoredNodes = null No live nodes contain current block Block locations: Dead nodes: . Will get new block locations from namenode and retry...
2018-06-15 14:12:41,119 WARN [vmbdsiwbdn2:16000.activeMasterManager] hdfs.DFSClient: DFS chooseDataNode: got # 3 IOException, will wait for 11833.399457334988 msec.
2018-06-15 14:12:52,955 WARN [vmbdsiwbdn2:16000.activeMasterManager] hdfs.DFSClient: Could not obtain block: BP-1307428289-10.0.0.4-1528240888625:blk_1073741830_1006 file=/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001 No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
2018-06-15 14:12:52,955 WARN [vmbdsiwbdn2:16000.activeMasterManager] hdfs.DFSClient: Could not obtain block: BP-1307428289-10.0.0.4-1528240888625:blk_1073741830_1006 file=/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001 No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException
2018-06-15 14:12:52,956 WARN [vmbdsiwbdn2:16000.activeMasterManager] hdfs.DFSClient: DFS Read
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1307428289-10.0.0.4-1528240888625:blk_1073741830_1006 file=/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-06-15 14:12:52,958 FATAL [vmbdsiwbdn2:16000.activeMasterManager] master.HMaster: Failed to become active master
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1307428289-10.0.0.4-1528240888625:blk_1073741830_1006 file=/apps/hbase/data/data/hbase/meta/.tabledesc/.tableinfo.0000000001 ---------------------------------------------------------------------------------------------- Please suggest how to overcome this scenario.
... View more
Labels:
- Labels:
-
Apache HBase
05-08-2018
10:44 AM
I have also received the same error. 2018-05-03 15:23:01,869 ERROR [xxxxxxxx.eastus.cloudapp.azure.com,16000,1525360743919_ChoreService_1] master.BackupLogCleaner: Failed to get hbase:backup table, therefore will keep all files I've observed that upon start/ restart, Active HBase Master are turning into Secondary HBase Master and then getting Stopped. Garbage Collection log is showing below message: Memory: 4k page, physical 8157984k(4535280k free), swap 0k(0k free)
CommandLine flags: -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -XX:InitialHeapSize=130527744 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=174485504 -XX:MaxTenuringThreshold=6 -XX:OldPLABSize=16 -XX:OnOutOfMemoryError=kill -9 %p -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-05-08T09:59:21.106+0000: 0.786: [GC (Allocation Failure) 2018-05-08T09:59:21.106+0000: 0.786: [ParNew: 34432K->4288K(38720K), 0.0066260 secs] 34432K->5497K(124736K), 0.0067169 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2018-05-08T09:59:21.819+0000: 1.500: [GC (Allocation Failure) 2018-05-08T09:59:21.819+0000: 1.500: [ParNew: 38720K->3327K(38720K), 0.0128638 secs] 39929K->6553K(124736K), 0.0129458 secs] [Times: user=0.02 sys=0.01, real=0.01 secs]
2018-05-08T09:59:22.098+0000: 1.778: [GC (Allocation Failure) 2018-05-08T09:59:22.098+0000: 1.778: [ParNew: 37759K->4287K(38720K), 0.0067072 secs] 40985K->10042K(124736K), 0.0067618 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-05-08T09:59:22.457+0000: 2.137: [GC (Allocation Failure) 2018-05-08T09:59:22.457+0000: 2.137: [ParNew: 38719K->3036K(38720K), 0.0071463 secs] 44474K->9248K(124736K), 0.0072080 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] Please share your comments if you've already found out the resolution for this issue.
... View more