Support Questions

Find answers, ask questions, and share your expertise

HBase RegionServer stop automatically

avatar
New Contributor

When I run QuickStart VMS for VMWARE on 12, HBase RegionServer stop automatically, please help me!
Here are logs,
hbase-hbase-regionserver-quickstart.cloudera.log:

 

2017-06-06 22:21:36,326 WARN  [ResponseProcessor for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742697_1873] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742697_1873
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1075)
2017-06-06 22:21:37,355 INFO  [quickstart.cloudera,60020,1496810964011_ChoreService_1] hbase.ScheduledChore: Chore: quickstart.cloudera,60020,1496810964011-MemstoreFlusherChore missed its start time
2017-06-06 22:21:37,356 INFO  [quickstart.cloudera,60020,1496810964011_ChoreService_2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time
2017-06-06 22:21:40,490 WARN  [ResponseProcessor for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742698_1874] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742698_1874
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1075)
2017-06-06 22:21:59,870 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] util.Sleeper: We slept 81086ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2017-06-06 22:22:00,095 INFO  [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 81657ms for sessionid 0x15c80e11a9b0004, closing socket connection and attempting reconnect
2017-06-06 22:22:00,099 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x15c80e11a9b0005, likely server has closed socket, closing socket connection and attempting reconnect
2017-06-06 22:22:01,682 INFO  [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-06 22:22:01,682 INFO  [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51079, server: quickstart.cloudera/127.0.0.1:2181
2017-06-06 22:22:01,689 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region server quickstart.cloudera,60020,1496810964011: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase regionserver:60020-0x15c80e11a9b0004 received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:700)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:611)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2017-06-06 22:22:01,689 INFO  [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x15c80e11a9b0004 has expired, closing socket connection
2017-06-06 22:22:01,690 FATAL [main-EventThread] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
2017-06-06 22:22:01,776 INFO  [main-EventThread] regionserver.HRegionServer: Dump of metrics as JSON on abort: {
  "beans" : [ {
    "name" : "java.lang:type=Memory",
    "modelerType" : "sun.management.MemoryImpl",
    "NonHeapMemoryUsage" : {
      "committed" : 136773632,
      "init" : 136773632,
      "max" : 184549376,
      "used" : 45819504
    },
    "ObjectPendingFinalizationCount" : 0,
    "HeapMemoryUsage" : {
      "committed" : 76742656,
      "init" : 79339968,
      "max" : 1253441536,
      "used" : 39087448
    },
    "Verbose" : false,
    "ObjectName" : "java.lang:type=Memory"
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC",
    "modelerType" : "RegionServer,sub=IPC",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "quickstart.cloudera",
    "queueSize" : 0,
    "numCallsInGeneralQueue" : 0,
    "numCallsInReplicationQueue" : 0,
    "numCallsInPriorityQueue" : 0,
    "numOpenConnections" : 1,
    "numActiveHandler" : 0,
    "TotalCallTime_num_ops" : 46,
    "TotalCallTime_min" : 0,
    "TotalCallTime_max" : 7648,
    "TotalCallTime_mean" : 186,
    "TotalCallTime_25th_percentile" : 300,
    "TotalCallTime_median" : 600,
    "TotalCallTime_75th_percentile" : 900,
    "TotalCallTime_90th_percentile" : 1081,
    "TotalCallTime_95th_percentile" : 1141,
    "TotalCallTime_98th_percentile" : 7099,
    "TotalCallTime_99th_percentile" : 7373,
    "TotalCallTime_99.9th_percentile" : 7620,
    "TotalCallTime_TimeRangeCount_0-1" : 45,
    "TotalCallTime_TimeRangeCount_3000-10000" : 1,
    "exceptions.FailedSanityCheckException" : 0,
    "exceptions.RegionMovedException" : 0,
    "QueueCallTime_num_ops" : 46,
    "QueueCallTime_min" : 0,
    "QueueCallTime_max" : 130,
    "QueueCallTime_mean" : 3,
    "QueueCallTime_25th_percentile" : 32,
    "QueueCallTime_median" : 65,
    "QueueCallTime_75th_percentile" : 97,
    "QueueCallTime_90th_percentile" : 117,
    "QueueCallTime_95th_percentile" : 123,
    "QueueCallTime_98th_percentile" : 127,
    "QueueCallTime_99th_percentile" : 128,
    "QueueCallTime_99.9th_percentile" : 129,
    "QueueCallTime_TimeRangeCount_0-1" : 46,
    "authenticationFailures" : 0,
    "authorizationFailures" : 0,
    "authenticationFallbacks" : 0,
    "exceptions" : 0,
    "RequestSize_num_ops" : 46,
    "RequestSize_min" : 12,
    "RequestSize_max" : 109,
    "RequestSize_mean" : 41,
    "RequestSize_25th_percentile" : 36,
    "RequestSize_median" : 60,
    "RequestSize_75th_percentile" : 84,
    "RequestSize_90th_percentile" : 99,
    "RequestSize_95th_percentile" : 104,
    "RequestSize_98th_percentile" : 107,
    "RequestSize_99th_percentile" : 108,
    "RequestSize_99.9th_percentile" : 108,
    "RequestSize_SizeRangeCount_0-10" : 46,
    "ResponseSize_num_ops" : 46,
    "ResponseSize_min" : 2,
    "ResponseSize_max" : 485,
    "ResponseSize_mean" : 18,
    "ResponseSize_25th_percentile" : 122,
    "ResponseSize_median" : 243,
    "ResponseSize_75th_percentile" : 364,
    "ResponseSize_90th_percentile" : 436,
    "ResponseSize_95th_percentile" : 460,
    "ResponseSize_98th_percentile" : 475,
    "ResponseSize_99th_percentile" : 480,
    "ResponseSize_99.9th_percentile" : 484,
    "ResponseSize_SizeRangeCount_0-10" : 46,
    "authenticationSuccesses" : 0,
    "authorizationSuccesses" : 8,
    "ProcessCallTime_num_ops" : 46,
    "ProcessCallTime_min" : 0,
    "ProcessCallTime_max" : 7518,
    "ProcessCallTime_mean" : 183,
    "ProcessCallTime_25th_percentile" : 300,
    "ProcessCallTime_median" : 600,
    "ProcessCallTime_75th_percentile" : 900,
    "ProcessCallTime_90th_percentile" : 1081,
    "ProcessCallTime_95th_percentile" : 1141,
    "ProcessCallTime_98th_percentile" : 7089,
    "ProcessCallTime_99th_percentile" : 7303,
    "ProcessCallTime_99.9th_percentile" : 7496,
    "ProcessCallTime_TimeRangeCount_0-1" : 45,
    "ProcessCallTime_TimeRangeCount_3000-10000" : 1,
    "exceptions.NotServingRegionException" : 0,
    "sentBytes" : 4320,
    "exceptions.RegionTooBusyException" : 0,
    "receivedBytes" : 5141,
    "exceptions.OutOfOrderScannerNextException" : 0,
    "exceptions.multiResponseTooLarge" : 0,
    "exceptions.UnknownScannerException" : 0
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication",
    "modelerType" : "RegionServer,sub=Replication",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "quickstart.cloudera",
    "sink.appliedOps" : 0,
    "sink.appliedBatches" : 0,
    "sink.ageOfLastAppliedOp" : 0
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server",
    "modelerType" : "RegionServer,sub=Server",
    "tag.zookeeperQuorum" : "localhost:2181",
    "tag.serverName" : "quickstart.cloudera,60020,1496810964011",
    "tag.clusterId" : "ae292c1c-713f-4dbe-8a41-7c6e94a1a0e6",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "quickstart.cloudera",
    "regionCount" : 2,
    "storeCount" : 2,
    "hlogFileCount" : 2,
    "hlogFileSize" : 0,
    "storeFileCount" : 2,
    "memStoreSize" : 864,
    "storeFileSize" : 2461,
    "regionServerStartTime" : 1496810964011,
    "totalRequestCount" : 47,
    "readRequestCount" : 14,
    "writeRequestCount" : 4,
    "checkMutateFailedCount" : 0,
    "checkMutatePassedCount" : 0,
    "storeFileIndexSize" : 856,
    "staticIndexSize" : 140,
    "staticBloomSize" : 4,
    "mutationsWithoutWALCount" : 0,
    "mutationsWithoutWALSize" : 0,
    "percentFilesLocal" : 100.0,
    "percentFilesLocalSecondaryRegions" : 0.0,
    "splitQueueLength" : 0,
    "compactionQueueLength" : 0,
    "flushQueueLength" : 0,
    "blockCacheFreeSize" : 500860056,
    "blockCacheCount" : 1,
    "blockCacheSize" : 516552,
    "blockCacheHitCount" : 4,
    "blockCacheHitCountPrimary" : 4,
    "blockCacheMissCount" : 1,
    "blockCacheMissCountPrimary" : 1,
    "blockCacheEvictionCount" : 0,
    "blockCacheEvictionCountPrimary" : 0,
    "blockCacheCountHitPercent" : 80.0000011920929,
    "blockCountHitPercent" : 80,
    "blockCacheExpres**bleep**Percent" : 80.0000011920929,
    "blockCacheFailedInsertionCount" : 0,
    "updatesBlockedTime" : 0,
    "flushedCellsCount" : 6,
    "compactedCellsCount" : 0,
    "majorCompactedCellsCount" : 0,
    "flushedCellsSize" : 1344,
    "compactedCellsSize" : 0,
    "majorCompactedCellsSize" : 0,
    "blockedRequestCount" : 0,
    "hedgedReads" : 0,
    "hedgedReadWins" : 0,
    "mobCompactedFromMobCellsCount" : 0,
    "mobCompactedIntoMobCellsCount" : 0,
    "mobCompactedFromMobCellsSize" : 0,
    "mobCompactedIntoMobCellsSize" : 0,
    "mobFlushCount" : 0,
    "mobFlushedCellsCount" : 0,
    "mobFlushedCellsSize" : 0,
    "mobScanCellsCount" : 0,
    "mobScanCellsSize" : 0,
    "mobFileCacheCount" : 0,
    "mobFileCacheAccessCount" : 0,
    "mobFileCacheMissCount" : 0,
    "mobFileCacheEvictedCount" : 0,
    "mobFileCacheHitPercent" : 0,
    "PauseTimeWithGc_num_ops" : 0,
    "PauseTimeWithGc_min" : 0,
    "PauseTimeWithGc_max" : 0,
    "PauseTimeWithGc_mean" : 0,
    "PauseTimeWithGc_25th_percentile" : 0,
    "PauseTimeWithGc_median" : 0,
    "PauseTimeWithGc_75th_percentile" : 0,
    "PauseTimeWithGc_90th_percentile" : 0,
    "PauseTimeWithGc_95th_percentile" : 0,
    "PauseTimeWithGc_98th_percentile" : 0,
    "PauseTimeWithGc_99th_percentile" : 0,
    "PauseTimeWithGc_99.9th_percentile" : 0,
    "pauseWarnThresholdExceeded" : 0,
    "splitSuccessCount" : 0,
    "splitRequestCount" : 0,
    "Append_num_ops" : 0,
    "Append_min" : 0,
    "Append_max" : 0,
    "Append_mean" : 0,
    "Append_25th_percentile" : 0,
    "Append_median" : 0,
    "Append_75th_percentile" : 0,
    "Append_90th_percentile" : 0,
    "Append_95th_percentile" : 0,
    "Append_98th_percentile" : 0,
    "Append_99th_percentile" : 0,
    "Append_99.9th_percentile" : 0,
    "Delete_num_ops" : 0,
    "Delete_min" : 0,
    "Delete_max" : 0,
    "Delete_mean" : 0,
    "Delete_25th_percentile" : 0,
    "Delete_median" : 0,
    "Delete_75th_percentile" : 0,
    "Delete_90th_percentile" : 0,
    "Delete_95th_percentile" : 0,
    "Delete_98th_percentile" : 0,
    "Delete_99th_percentile" : 0,
    "Delete_99.9th_percentile" : 0,
    "Mutate_num_ops" : 4,
    "Mutate_min" : 4,
    "Mutate_max" : 56,
    "Mutate_mean" : 18,
    "Mutate_25th_percentile" : 17,
    "Mutate_median" : 30,
    "Mutate_75th_percentile" : 43,
    "Mutate_90th_percentile" : 50,
    "Mutate_95th_percentile" : 53,
    "Mutate_98th_percentile" : 54,
    "Mutate_99th_percentile" : 55,
    "Mutate_99.9th_percentile" : 55,
    "Mutate_TimeRangeCount_0-1" : 4,
    "ScanNext_num_ops" : 12,
    "ScanNext_min" : 0,
    "ScanNext_max" : 744,
    "ScanNext_mean" : 390,
    "ScanNext_25th_percentile" : 186,
    "ScanNext_median" : 372,
    "ScanNext_75th_percentile" : 558,
    "ScanNext_90th_percentile" : 669,
    "ScanNext_95th_percentile" : 706,
    "ScanNext_98th_percentile" : 729,
    "ScanNext_99th_percentile" : 736,
    "ScanNext_99.9th_percentile" : 743,
    "ScanNext_TimeRangeCount_0-1" : 12,
    "slowDeleteCount" : 0,
    "slowIncrementCount" : 0,
    "FlushTime_num_ops" : 2,
    "FlushTime_min" : 9574,
    "FlushTime_max" : 193759,
    "FlushTime_mean" : 101666,
    "FlushTime_25th_percentile" : 10076,
    "FlushTime_median" : 10578,
    "FlushTime_75th_percentile" : 193260,
    "FlushTime_90th_percentile" : 193559,
    "FlushTime_95th_percentile" : 193659,
    "FlushTime_98th_percentile" : 193719,
    "FlushTime_99th_percentile" : 193739,
    "FlushTime_99.9th_percentile" : 193757,
    "FlushTime_TimeRangeCount_3000-10000" : 1,
    "FlushTime_TimeRangeCount_120000-300000" : 1,
    "Get_num_ops" : 5,
    "Get_min" : 0,
    "Get_max" : 55,
    "Get_mean" : 11,
    "Get_25th_percentile" : 13,
    "Get_median" : 27,
    "Get_75th_percentile" : 41,
    "Get_90th_percentile" : 49,
    "Get_95th_percentile" : 52,
    "Get_98th_percentile" : 53,
    "Get_99th_percentile" : 54,
    "Get_99.9th_percentile" : 54,
    "Get_TimeRangeCount_0-1" : 5,
    "Replay_num_ops" : 0,
    "Replay_min" : 0,
    "Replay_max" : 0,
    "Replay_mean" : 0,
    "Replay_25th_percentile" : 0,
    "Replay_median" : 0,
    "Replay_75th_percentile" : 0,
    "Replay_90th_percentile" : 0,
    "Replay_95th_percentile" : 0,
    "Replay_98th_percentile" : 0,
    "Replay_99th_percentile" : 0,
    "Replay_99.9th_percentile" : 0,
    "slowGetCount" : 0,
    "slowAppendCount" : 0,
    "slowPutCount" : 0,
    "PauseTimeWithoutGc_num_ops" : 0,
    "PauseTimeWithoutGc_min" : 0,
    "PauseTimeWithoutGc_max" : 0,
    "PauseTimeWithoutGc_mean" : 0,
    "PauseTimeWithoutGc_25th_percentile" : 0,
    "PauseTimeWithoutGc_median" : 0,
    "PauseTimeWithoutGc_75th_percentile" : 0,
    "PauseTimeWithoutGc_90th_percentile" : 0,
    "PauseTimeWithoutGc_95th_percentile" : 0,
    "PauseTimeWithoutGc_98th_percentile" : 0,
    "PauseTimeWithoutGc_99th_percentile" : 0,
    "PauseTimeWithoutGc_99.9th_percentile" : 0,
    "SplitTime_num_ops" : 0,
    "SplitTime_min" : 0,
    "SplitTime_max" : 0,
    "SplitTime_mean" : 0,
    "SplitTime_25th_percentile" : 0,
    "SplitTime_median" : 0,
    "SplitTime_75th_percentile" : 0,
    "SplitTime_90th_percentile" : 0,
    "SplitTime_95th_percentile" : 0,
    "SplitTime_98th_percentile" : 0,
    "SplitTime_99th_percentile" : 0,
    "SplitTime_99.9th_percentile" : 0,
    "pauseInfoThresholdExceeded" : 0,
    "Increment_num_ops" : 0,
    "Increment_min" : 0,
    "Increment_max" : 0,
    "Increment_mean" : 0,
    "Increment_25th_percentile" : 0,
    "Increment_median" : 0,
    "Increment_75th_percentile" : 0,
    "Increment_90th_percentile" : 0,
    "Increment_95th_percentile" : 0,
    "Increment_98th_percentile" : 0,
    "Increment_99th_percentile" : 0,
    "Increment_99.9th_percentile" : 0
  } ]
}
2017-06-06 22:22:01,801 INFO  [main-EventThread] regionserver.HRegionServer: STOPPED: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase regionserver:60020-0x15c80e11a9b0004 received expired from ZooKeeper, aborting
2017-06-06 22:22:01,801 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2017-06-06 22:22:01,801 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Stopping infoServer
2017-06-06 22:22:01,805 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-06-06 22:22:01,806 INFO  [SplitLogWorker-quickstart:60020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting. 
2017-06-06 22:22:01,823 INFO  [SplitLogWorker-quickstart:60020] regionserver.SplitLogWorker: SplitLogWorker quickstart.cloudera,60020,1496810964011 exiting
2017-06-06 22:22:01,827 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
2017-06-06 22:22:01,832 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore.
2017-06-06 22:22:01,833 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2017-06-06 22:22:01,834 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2017-06-06 22:22:01,832 INFO  [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2017-06-06 22:22:01,832 INFO  [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2017-06-06 22:22:01,837 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: aborting server quickstart.cloudera,60020,1496810964011
2017-06-06 22:22:01,838 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15c80e11a9b0005
2017-06-06 22:22:01,843 INFO  [StoreCloserThread-hbase:namespace,,1496811020881.6e8c9b0e20974475891f01654c517d9b.-1] regionserver.HStore: Closed info
2017-06-06 22:22:01,848 INFO  [RS_CLOSE_REGION-quickstart:60020-0] regionserver.HRegion: Closed hbase:namespace,,1496811020881.6e8c9b0e20974475891f01654c517d9b.
2017-06-06 22:22:01,910 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-06 22:22:01,911 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51081, server: quickstart.cloudera/127.0.0.1:2181
2017-06-06 22:22:02,014 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZooKeeper: Session: 0x15c80e11a9b0005 closed
2017-06-06 22:22:02,015 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-06-06 22:22:02,015 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2017-06-06 22:22:02,015 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2017-06-06 22:22:02,015 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2017-06-06 22:22:02,016 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2017-06-06 22:22:02,023 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Waiting on 1 regions to close
2017-06-06 22:22:02,031 INFO  [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore: Closed info
2017-06-06 22:22:02,033 INFO  [RS_CLOSE_META-quickstart:60020-0] regionserver.HRegion: Closed hbase:meta,,1.1588230740
2017-06-06 22:22:02,224 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: stopping server quickstart.cloudera,60020,1496810964011; all regions closed.
2017-06-06 22:22:02,225 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing...
java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
2017-06-06 22:22:02,227 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing...
java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
2017-06-06 22:22:02,230 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Shutdown / close of WAL failed: java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting...
2017-06-06 22:22:02,242 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020 closing leases
2017-06-06 22:22:02,242 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020 closed leases
2017-06-06 22:22:02,243 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] hbase.ChoreService: Chore service for: quickstart.cloudera,60020,1496810964011 had [[ScheduledChore: Name: quickstart.cloudera,60020,1496810964011-MemstoreFlusherChore Period: 10000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region quickstart.cloudera,60020,1496810964011 Period: 120000 Unit: MILLISECONDS]] on shutdown
2017-06-06 22:22:06,656 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020.logRoller] regionserver.LogRoller: LogRoller exiting.
2017-06-06 22:22:06,722 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker closing leases
2017-06-06 22:22:06,723 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker closed leases
2017-06-06 22:22:07,883 INFO  [RS_OPEN_META-quickstart:60020-0-MetaLogRoller] regionserver.LogRoller: LogRoller exiting.
2017-06-06 22:22:22,891 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts
2017-06-06 22:22:22,891 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZKUtil: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase Unable to list children of znode /hbase/replication/rs/quickstart.cloudera,60020,1496810964011 
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/quickstart.cloudera,60020,1496810964011
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
	at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2162)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088)
	at java.lang.Thread.run(Thread.java:745)
2017-06-06 22:22:22,894 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZooKeeperWatcher: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/quickstart.cloudera,60020,1496810964011
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
	at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2162)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088)
	at java.lang.Thread.run(Thread.java:745)
2017-06-06 22:22:22,896 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] ipc.RpcServer: Stopping server on 60020
2017-06-06 22:22:22,904 INFO  [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: stopping
2017-06-06 22:22:22,928 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2017-06-06 22:22:22,928 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2017-06-06 22:22:37,975 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2017-06-06 22:22:37,975 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/quickstart.cloudera,60020,1496810964011
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1236)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1225)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1427)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1096)
	at java.lang.Thread.run(Thread.java:745)
2017-06-06 22:22:37,978 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: stopping server quickstart.cloudera,60020,1496810964011; zookeeper connection closed.
2017-06-06 22:22:37,978 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: regionserver/quickstart.cloudera/127.0.0.1:60020 exiting
2017-06-06 22:22:37,981 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2676)
2017-06-06 22:22:37,996 INFO  [Thread-7] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@439b2c10
2017-06-06 22:22:37,996 INFO  [Thread-7] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2017-06-06 22:22:37,998 INFO  [Thread-7] regionserver.ShutdownHook: Shutdown hook finished.
3 REPLIES 3

avatar
Mentor
It seems like your VM either has too little RAM or is unable to get adequate CPU cycles to run the RS continually, per the below snippet:

2017-06-06 22:21:59,870 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] util.Sleeper: We slept 81086ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired

Can you try increasing the RAM available to the VM?

avatar
Explorer

did it finally resolved?

avatar
Super Collaborator

Hello @xgxshtc 

 

We observed you have posted the concerned ask in a New Post [1] as the concerned Post is ~4Years Old. While the Current Post is Unresolved, We shall wait on your Team's review on [1] before confirming the Solution on the Current Post as well.

 

Regards, Smarak

 

[1] https://community.cloudera.com/t5/Support-Questions/Hbase-regionserver-shutdown-after-few-hours/m-p/...