Reply
New Contributor
Posts: 1
Registered: ‎06-06-2017

HBase RegionServer stop automatically

When I run QuickStart VMS for VMWARE on 12, HBase RegionServer stop automatically, please help me!
Here are logs,
hbase-hbase-regionserver-quickstart.cloudera.log:

 

2017-06-06 22:21:36,326 WARN  [ResponseProcessor for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742697_1873] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742697_1873
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1075)
2017-06-06 22:21:37,355 INFO  [quickstart.cloudera,60020,1496810964011_ChoreService_1] hbase.ScheduledChore: Chore: quickstart.cloudera,60020,1496810964011-MemstoreFlusherChore missed its start time
2017-06-06 22:21:37,356 INFO  [quickstart.cloudera,60020,1496810964011_ChoreService_2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time
2017-06-06 22:21:40,490 WARN  [ResponseProcessor for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742698_1874] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742698_1874
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1075)
2017-06-06 22:21:59,870 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] util.Sleeper: We slept 81086ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2017-06-06 22:22:00,095 INFO  [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 81657ms for sessionid 0x15c80e11a9b0004, closing socket connection and attempting reconnect
2017-06-06 22:22:00,099 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x15c80e11a9b0005, likely server has closed socket, closing socket connection and attempting reconnect
2017-06-06 22:22:01,682 INFO  [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-06 22:22:01,682 INFO  [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51079, server: quickstart.cloudera/127.0.0.1:2181
2017-06-06 22:22:01,689 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region server quickstart.cloudera,60020,1496810964011: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase regionserver:60020-0x15c80e11a9b0004 received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:700)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:611)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2017-06-06 22:22:01,689 INFO  [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x15c80e11a9b0004 has expired, closing socket connection
2017-06-06 22:22:01,690 FATAL [main-EventThread] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
2017-06-06 22:22:01,776 INFO  [main-EventThread] regionserver.HRegionServer: Dump of metrics as JSON on abort: {
  "beans" : [ {
    "name" : "java.lang:type=Memory",
    "modelerType" : "sun.management.MemoryImpl",
    "NonHeapMemoryUsage" : {
      "committed" : 136773632,
      "init" : 136773632,
      "max" : 184549376,
      "used" : 45819504
    },
    "ObjectPendingFinalizationCount" : 0,
    "HeapMemoryUsage" : {
      "committed" : 76742656,
      "init" : 79339968,
      "max" : 1253441536,
      "used" : 39087448
    },
    "Verbose" : false,
    "ObjectName" : "java.lang:type=Memory"
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC",
    "modelerType" : "RegionServer,sub=IPC",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "quickstart.cloudera",
    "queueSize" : 0,
    "numCallsInGeneralQueue" : 0,
    "numCallsInReplicationQueue" : 0,
    "numCallsInPriorityQueue" : 0,
    "numOpenConnections" : 1,
    "numActiveHandler" : 0,
    "TotalCallTime_num_ops" : 46,
    "TotalCallTime_min" : 0,
    "TotalCallTime_max" : 7648,
    "TotalCallTime_mean" : 186,
    "TotalCallTime_25th_percentile" : 300,
    "TotalCallTime_median" : 600,
    "TotalCallTime_75th_percentile" : 900,
    "TotalCallTime_90th_percentile" : 1081,
    "TotalCallTime_95th_percentile" : 1141,
    "TotalCallTime_98th_percentile" : 7099,
    "TotalCallTime_99th_percentile" : 7373,
    "TotalCallTime_99.9th_percentile" : 7620,
    "TotalCallTime_TimeRangeCount_0-1" : 45,
    "TotalCallTime_TimeRangeCount_3000-10000" : 1,
    "exceptions.FailedSanityCheckException" : 0,
    "exceptions.RegionMovedException" : 0,
    "QueueCallTime_num_ops" : 46,
    "QueueCallTime_min" : 0,
    "QueueCallTime_max" : 130,
    "QueueCallTime_mean" : 3,
    "QueueCallTime_25th_percentile" : 32,
    "QueueCallTime_median" : 65,
    "QueueCallTime_75th_percentile" : 97,
    "QueueCallTime_90th_percentile" : 117,
    "QueueCallTime_95th_percentile" : 123,
    "QueueCallTime_98th_percentile" : 127,
    "QueueCallTime_99th_percentile" : 128,
    "QueueCallTime_99.9th_percentile" : 129,
    "QueueCallTime_TimeRangeCount_0-1" : 46,
    "authenticationFailures" : 0,
    "authorizationFailures" : 0,
    "authenticationFallbacks" : 0,
    "exceptions" : 0,
    "RequestSize_num_ops" : 46,
    "RequestSize_min" : 12,
    "RequestSize_max" : 109,
    "RequestSize_mean" : 41,
    "RequestSize_25th_percentile" : 36,
    "RequestSize_median" : 60,
    "RequestSize_75th_percentile" : 84,
    "RequestSize_90th_percentile" : 99,
    "RequestSize_95th_percentile" : 104,
    "RequestSize_98th_percentile" : 107,
    "RequestSize_99th_percentile" : 108,
    "RequestSize_99.9th_percentile" : 108,
    "RequestSize_SizeRangeCount_0-10" : 46,
    "ResponseSize_num_ops" : 46,
    "ResponseSize_min" : 2,
    "ResponseSize_max" : 485,
    "ResponseSize_mean" : 18,
    "ResponseSize_25th_percentile" : 122,
    "ResponseSize_median" : 243,
    "ResponseSize_75th_percentile" : 364,
    "ResponseSize_90th_percentile" : 436,
    "ResponseSize_95th_percentile" : 460,
    "ResponseSize_98th_percentile" : 475,
    "ResponseSize_99th_percentile" : 480,
    "ResponseSize_99.9th_percentile" : 484,
    "ResponseSize_SizeRangeCount_0-10" : 46,
    "authenticationSuccesses" : 0,
    "authorizationSuccesses" : 8,
    "ProcessCallTime_num_ops" : 46,
    "ProcessCallTime_min" : 0,
    "ProcessCallTime_max" : 7518,
    "ProcessCallTime_mean" : 183,
    "ProcessCallTime_25th_percentile" : 300,
    "ProcessCallTime_median" : 600,
    "ProcessCallTime_75th_percentile" : 900,
    "ProcessCallTime_90th_percentile" : 1081,
    "ProcessCallTime_95th_percentile" : 1141,
    "ProcessCallTime_98th_percentile" : 7089,
    "ProcessCallTime_99th_percentile" : 7303,
    "ProcessCallTime_99.9th_percentile" : 7496,
    "ProcessCallTime_TimeRangeCount_0-1" : 45,
    "ProcessCallTime_TimeRangeCount_3000-10000" : 1,
    "exceptions.NotServingRegionException" : 0,
    "sentBytes" : 4320,
    "exceptions.RegionTooBusyException" : 0,
    "receivedBytes" : 5141,
    "exceptions.OutOfOrderScannerNextException" : 0,
    "exceptions.multiResponseTooLarge" : 0,
    "exceptions.UnknownScannerException" : 0
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication",
    "modelerType" : "RegionServer,sub=Replication",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "quickstart.cloudera",
    "sink.appliedOps" : 0,
    "sink.appliedBatches" : 0,
    "sink.ageOfLastAppliedOp" : 0
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server",
    "modelerType" : "RegionServer,sub=Server",
    "tag.zookeeperQuorum" : "localhost:2181",
    "tag.serverName" : "quickstart.cloudera,60020,1496810964011",
    "tag.clusterId" : "ae292c1c-713f-4dbe-8a41-7c6e94a1a0e6",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "quickstart.cloudera",
    "regionCount" : 2,
    "storeCount" : 2,
    "hlogFileCount" : 2,
    "hlogFileSize" : 0,
    "storeFileCount" : 2,
    "memStoreSize" : 864,
    "storeFileSize" : 2461,
    "regionServerStartTime" : 1496810964011,
    "totalRequestCount" : 47,
    "readRequestCount" : 14,
    "writeRequestCount" : 4,
    "checkMutateFailedCount" : 0,
    "checkMutatePassedCount" : 0,
    "storeFileIndexSize" : 856,
    "staticIndexSize" : 140,
    "staticBloomSize" : 4,
    "mutationsWithoutWALCount" : 0,
    "mutationsWithoutWALSize" : 0,
    "percentFilesLocal" : 100.0,
    "percentFilesLocalSecondaryRegions" : 0.0,
    "splitQueueLength" : 0,
    "compactionQueueLength" : 0,
    "flushQueueLength" : 0,
    "blockCacheFreeSize" : 500860056,
    "blockCacheCount" : 1,
    "blockCacheSize" : 516552,
    "blockCacheHitCount" : 4,
    "blockCacheHitCountPrimary" : 4,
    "blockCacheMissCount" : 1,
    "blockCacheMissCountPrimary" : 1,
    "blockCacheEvictionCount" : 0,
    "blockCacheEvictionCountPrimary" : 0,
    "blockCacheCountHitPercent" : 80.0000011920929,
    "blockCountHitPercent" : 80,
    "blockCacheExpres**bleep**Percent" : 80.0000011920929,
    "blockCacheFailedInsertionCount" : 0,
    "updatesBlockedTime" : 0,
    "flushedCellsCount" : 6,
    "compactedCellsCount" : 0,
    "majorCompactedCellsCount" : 0,
    "flushedCellsSize" : 1344,
    "compactedCellsSize" : 0,
    "majorCompactedCellsSize" : 0,
    "blockedRequestCount" : 0,
    "hedgedReads" : 0,
    "hedgedReadWins" : 0,
    "mobCompactedFromMobCellsCount" : 0,
    "mobCompactedIntoMobCellsCount" : 0,
    "mobCompactedFromMobCellsSize" : 0,
    "mobCompactedIntoMobCellsSize" : 0,
    "mobFlushCount" : 0,
    "mobFlushedCellsCount" : 0,
    "mobFlushedCellsSize" : 0,
    "mobScanCellsCount" : 0,
    "mobScanCellsSize" : 0,
    "mobFileCacheCount" : 0,
    "mobFileCacheAccessCount" : 0,
    "mobFileCacheMissCount" : 0,
    "mobFileCacheEvictedCount" : 0,
    "mobFileCacheHitPercent" : 0,
    "PauseTimeWithGc_num_ops" : 0,
    "PauseTimeWithGc_min" : 0,
    "PauseTimeWithGc_max" : 0,
    "PauseTimeWithGc_mean" : 0,
    "PauseTimeWithGc_25th_percentile" : 0,
    "PauseTimeWithGc_median" : 0,
    "PauseTimeWithGc_75th_percentile" : 0,
    "PauseTimeWithGc_90th_percentile" : 0,
    "PauseTimeWithGc_95th_percentile" : 0,
    "PauseTimeWithGc_98th_percentile" : 0,
    "PauseTimeWithGc_99th_percentile" : 0,
    "PauseTimeWithGc_99.9th_percentile" : 0,
    "pauseWarnThresholdExceeded" : 0,
    "splitSuccessCount" : 0,
    "splitRequestCount" : 0,
    "Append_num_ops" : 0,
    "Append_min" : 0,
    "Append_max" : 0,
    "Append_mean" : 0,
    "Append_25th_percentile" : 0,
    "Append_median" : 0,
    "Append_75th_percentile" : 0,
    "Append_90th_percentile" : 0,
    "Append_95th_percentile" : 0,
    "Append_98th_percentile" : 0,
    "Append_99th_percentile" : 0,
    "Append_99.9th_percentile" : 0,
    "Delete_num_ops" : 0,
    "Delete_min" : 0,
    "Delete_max" : 0,
    "Delete_mean" : 0,
    "Delete_25th_percentile" : 0,
    "Delete_median" : 0,
    "Delete_75th_percentile" : 0,
    "Delete_90th_percentile" : 0,
    "Delete_95th_percentile" : 0,
    "Delete_98th_percentile" : 0,
    "Delete_99th_percentile" : 0,
    "Delete_99.9th_percentile" : 0,
    "Mutate_num_ops" : 4,
    "Mutate_min" : 4,
    "Mutate_max" : 56,
    "Mutate_mean" : 18,
    "Mutate_25th_percentile" : 17,
    "Mutate_median" : 30,
    "Mutate_75th_percentile" : 43,
    "Mutate_90th_percentile" : 50,
    "Mutate_95th_percentile" : 53,
    "Mutate_98th_percentile" : 54,
    "Mutate_99th_percentile" : 55,
    "Mutate_99.9th_percentile" : 55,
    "Mutate_TimeRangeCount_0-1" : 4,
    "ScanNext_num_ops" : 12,
    "ScanNext_min" : 0,
    "ScanNext_max" : 744,
    "ScanNext_mean" : 390,
    "ScanNext_25th_percentile" : 186,
    "ScanNext_median" : 372,
    "ScanNext_75th_percentile" : 558,
    "ScanNext_90th_percentile" : 669,
    "ScanNext_95th_percentile" : 706,
    "ScanNext_98th_percentile" : 729,
    "ScanNext_99th_percentile" : 736,
    "ScanNext_99.9th_percentile" : 743,
    "ScanNext_TimeRangeCount_0-1" : 12,
    "slowDeleteCount" : 0,
    "slowIncrementCount" : 0,
    "FlushTime_num_ops" : 2,
    "FlushTime_min" : 9574,
    "FlushTime_max" : 193759,
    "FlushTime_mean" : 101666,
    "FlushTime_25th_percentile" : 10076,
    "FlushTime_median" : 10578,
    "FlushTime_75th_percentile" : 193260,
    "FlushTime_90th_percentile" : 193559,
    "FlushTime_95th_percentile" : 193659,
    "FlushTime_98th_percentile" : 193719,
    "FlushTime_99th_percentile" : 193739,
    "FlushTime_99.9th_percentile" : 193757,
    "FlushTime_TimeRangeCount_3000-10000" : 1,
    "FlushTime_TimeRangeCount_120000-300000" : 1,
    "Get_num_ops" : 5,
    "Get_min" : 0,
    "Get_max" : 55,
    "Get_mean" : 11,
    "Get_25th_percentile" : 13,
    "Get_median" : 27,
    "Get_75th_percentile" : 41,
    "Get_90th_percentile" : 49,
    "Get_95th_percentile" : 52,
    "Get_98th_percentile" : 53,
    "Get_99th_percentile" : 54,
    "Get_99.9th_percentile" : 54,
    "Get_TimeRangeCount_0-1" : 5,
    "Replay_num_ops" : 0,
    "Replay_min" : 0,
    "Replay_max" : 0,
    "Replay_mean" : 0,
    "Replay_25th_percentile" : 0,
    "Replay_median" : 0,
    "Replay_75th_percentile" : 0,
    "Replay_90th_percentile" : 0,
    "Replay_95th_percentile" : 0,
    "Replay_98th_percentile" : 0,
    "Replay_99th_percentile" : 0,
    "Replay_99.9th_percentile" : 0,
    "slowGetCount" : 0,
    "slowAppendCount" : 0,
    "slowPutCount" : 0,
    "PauseTimeWithoutGc_num_ops" : 0,
    "PauseTimeWithoutGc_min" : 0,
    "PauseTimeWithoutGc_max" : 0,
    "PauseTimeWithoutGc_mean" : 0,
    "PauseTimeWithoutGc_25th_percentile" : 0,
    "PauseTimeWithoutGc_median" : 0,
    "PauseTimeWithoutGc_75th_percentile" : 0,
    "PauseTimeWithoutGc_90th_percentile" : 0,
    "PauseTimeWithoutGc_95th_percentile" : 0,
    "PauseTimeWithoutGc_98th_percentile" : 0,
    "PauseTimeWithoutGc_99th_percentile" : 0,
    "PauseTimeWithoutGc_99.9th_percentile" : 0,
    "SplitTime_num_ops" : 0,
    "SplitTime_min" : 0,
    "SplitTime_max" : 0,
    "SplitTime_mean" : 0,
    "SplitTime_25th_percentile" : 0,
    "SplitTime_median" : 0,
    "SplitTime_75th_percentile" : 0,
    "SplitTime_90th_percentile" : 0,
    "SplitTime_95th_percentile" : 0,
    "SplitTime_98th_percentile" : 0,
    "SplitTime_99th_percentile" : 0,
    "SplitTime_99.9th_percentile" : 0,
    "pauseInfoThresholdExceeded" : 0,
    "Increment_num_ops" : 0,
    "Increment_min" : 0,
    "Increment_max" : 0,
    "Increment_mean" : 0,
    "Increment_25th_percentile" : 0,
    "Increment_median" : 0,
    "Increment_75th_percentile" : 0,
    "Increment_90th_percentile" : 0,
    "Increment_95th_percentile" : 0,
    "Increment_98th_percentile" : 0,
    "Increment_99th_percentile" : 0,
    "Increment_99.9th_percentile" : 0
  } ]
}
2017-06-06 22:22:01,801 INFO  [main-EventThread] regionserver.HRegionServer: STOPPED: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase regionserver:60020-0x15c80e11a9b0004 received expired from ZooKeeper, aborting
2017-06-06 22:22:01,801 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2017-06-06 22:22:01,801 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Stopping infoServer
2017-06-06 22:22:01,805 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-06-06 22:22:01,806 INFO  [SplitLogWorker-quickstart:60020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting. 
2017-06-06 22:22:01,823 INFO  [SplitLogWorker-quickstart:60020] regionserver.SplitLogWorker: SplitLogWorker quickstart.cloudera,60020,1496810964011 exiting
2017-06-06 22:22:01,827 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
2017-06-06 22:22:01,832 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore.
2017-06-06 22:22:01,833 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2017-06-06 22:22:01,834 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2017-06-06 22:22:01,832 INFO  [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2017-06-06 22:22:01,832 INFO  [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2017-06-06 22:22:01,837 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: aborting server quickstart.cloudera,60020,1496810964011
2017-06-06 22:22:01,838 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15c80e11a9b0005
2017-06-06 22:22:01,843 INFO  [StoreCloserThread-hbase:namespace,,1496811020881.6e8c9b0e20974475891f01654c517d9b.-1] regionserver.HStore: Closed info
2017-06-06 22:22:01,848 INFO  [RS_CLOSE_REGION-quickstart:60020-0] regionserver.HRegion: Closed hbase:namespace,,1496811020881.6e8c9b0e20974475891f01654c517d9b.
2017-06-06 22:22:01,910 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-06 22:22:01,911 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51081, server: quickstart.cloudera/127.0.0.1:2181
2017-06-06 22:22:02,014 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZooKeeper: Session: 0x15c80e11a9b0005 closed
2017-06-06 22:22:02,015 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-06-06 22:22:02,015 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2017-06-06 22:22:02,015 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2017-06-06 22:22:02,015 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2017-06-06 22:22:02,016 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2017-06-06 22:22:02,023 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Waiting on 1 regions to close
2017-06-06 22:22:02,031 INFO  [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore: Closed info
2017-06-06 22:22:02,033 INFO  [RS_CLOSE_META-quickstart:60020-0] regionserver.HRegion: Closed hbase:meta,,1.1588230740
2017-06-06 22:22:02,224 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: stopping server quickstart.cloudera,60020,1496810964011; all regions closed.
2017-06-06 22:22:02,225 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing...
java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
2017-06-06 22:22:02,227 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing...
java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
2017-06-06 22:22:02,230 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Shutdown / close of WAL failed: java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting...
2017-06-06 22:22:02,242 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020 closing leases
2017-06-06 22:22:02,242 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020 closed leases
2017-06-06 22:22:02,243 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] hbase.ChoreService: Chore service for: quickstart.cloudera,60020,1496810964011 had [[ScheduledChore: Name: quickstart.cloudera,60020,1496810964011-MemstoreFlusherChore Period: 10000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region quickstart.cloudera,60020,1496810964011 Period: 120000 Unit: MILLISECONDS]] on shutdown
2017-06-06 22:22:06,656 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020.logRoller] regionserver.LogRoller: LogRoller exiting.
2017-06-06 22:22:06,722 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker closing leases
2017-06-06 22:22:06,723 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker closed leases
2017-06-06 22:22:07,883 INFO  [RS_OPEN_META-quickstart:60020-0-MetaLogRoller] regionserver.LogRoller: LogRoller exiting.
2017-06-06 22:22:22,891 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts
2017-06-06 22:22:22,891 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZKUtil: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase Unable to list children of znode /hbase/replication/rs/quickstart.cloudera,60020,1496810964011 
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/quickstart.cloudera,60020,1496810964011
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
	at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2162)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088)
	at java.lang.Thread.run(Thread.java:745)
2017-06-06 22:22:22,894 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZooKeeperWatcher: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/quickstart.cloudera,60020,1496810964011
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
	at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
	at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
	at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2162)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088)
	at java.lang.Thread.run(Thread.java:745)
2017-06-06 22:22:22,896 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] ipc.RpcServer: Stopping server on 60020
2017-06-06 22:22:22,904 INFO  [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: stopping
2017-06-06 22:22:22,928 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2017-06-06 22:22:22,928 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2017-06-06 22:22:37,975 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2017-06-06 22:22:37,975 WARN  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/quickstart.cloudera,60020,1496810964011
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1236)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1225)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1427)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1096)
	at java.lang.Thread.run(Thread.java:745)
2017-06-06 22:22:37,978 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: stopping server quickstart.cloudera,60020,1496810964011; zookeeper connection closed.
2017-06-06 22:22:37,978 INFO  [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: regionserver/quickstart.cloudera/127.0.0.1:60020 exiting
2017-06-06 22:22:37,981 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2676)
2017-06-06 22:22:37,996 INFO  [Thread-7] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@439b2c10
2017-06-06 22:22:37,996 INFO  [Thread-7] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2017-06-06 22:22:37,998 INFO  [Thread-7] regionserver.ShutdownHook: Shutdown hook finished.
Posts: 1,565
Kudos: 287
Solutions: 239
Registered: ‎07-31-2013

Re: HBase RegionServer stop automatically

It seems like your VM either has too little RAM or is unable to get adequate CPU cycles to run the RS continually, per the below snippet:

2017-06-06 22:21:59,870 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] util.Sleeper: We slept 81086ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired

Can you try increasing the RAM available to the VM?
Backline Customer Operations Engineer
Announcements