Member since
06-06-2017
1
Post
0
Kudos Received
0
Solutions
06-07-2017
12:21 AM
When I run QuickStart VMS for VMWARE on 12, HBase RegionServer stop automatically, please help me! Here are logs, hbase-hbase-regionserver-quickstart.cloudera.log: 2017-06-06 22:21:36,326 WARN [ResponseProcessor for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742697_1873] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742697_1873
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1075)
2017-06-06 22:21:37,355 INFO [quickstart.cloudera,60020,1496810964011_ChoreService_1] hbase.ScheduledChore: Chore: quickstart.cloudera,60020,1496810964011-MemstoreFlusherChore missed its start time
2017-06-06 22:21:37,356 INFO [quickstart.cloudera,60020,1496810964011_ChoreService_2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time
2017-06-06 22:21:40,490 WARN [ResponseProcessor for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742698_1874] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742698_1874
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1075)
2017-06-06 22:21:59,870 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] util.Sleeper: We slept 81086ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2017-06-06 22:22:00,095 INFO [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 81657ms for sessionid 0x15c80e11a9b0004, closing socket connection and attempting reconnect
2017-06-06 22:22:00,099 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x15c80e11a9b0005, likely server has closed socket, closing socket connection and attempting reconnect
2017-06-06 22:22:01,682 INFO [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-06 22:22:01,682 INFO [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51079, server: quickstart.cloudera/127.0.0.1:2181
2017-06-06 22:22:01,689 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region server quickstart.cloudera,60020,1496810964011: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase regionserver:60020-0x15c80e11a9b0004 received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:700)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:611)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2017-06-06 22:22:01,689 INFO [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x15c80e11a9b0004 has expired, closing socket connection
2017-06-06 22:22:01,690 FATAL [main-EventThread] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
2017-06-06 22:22:01,776 INFO [main-EventThread] regionserver.HRegionServer: Dump of metrics as JSON on abort: {
"beans" : [ {
"name" : "java.lang:type=Memory",
"modelerType" : "sun.management.MemoryImpl",
"NonHeapMemoryUsage" : {
"committed" : 136773632,
"init" : 136773632,
"max" : 184549376,
"used" : 45819504
},
"ObjectPendingFinalizationCount" : 0,
"HeapMemoryUsage" : {
"committed" : 76742656,
"init" : 79339968,
"max" : 1253441536,
"used" : 39087448
},
"Verbose" : false,
"ObjectName" : "java.lang:type=Memory"
} ],
"beans" : [ {
"name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC",
"modelerType" : "RegionServer,sub=IPC",
"tag.Context" : "regionserver",
"tag.Hostname" : "quickstart.cloudera",
"queueSize" : 0,
"numCallsInGeneralQueue" : 0,
"numCallsInReplicationQueue" : 0,
"numCallsInPriorityQueue" : 0,
"numOpenConnections" : 1,
"numActiveHandler" : 0,
"TotalCallTime_num_ops" : 46,
"TotalCallTime_min" : 0,
"TotalCallTime_max" : 7648,
"TotalCallTime_mean" : 186,
"TotalCallTime_25th_percentile" : 300,
"TotalCallTime_median" : 600,
"TotalCallTime_75th_percentile" : 900,
"TotalCallTime_90th_percentile" : 1081,
"TotalCallTime_95th_percentile" : 1141,
"TotalCallTime_98th_percentile" : 7099,
"TotalCallTime_99th_percentile" : 7373,
"TotalCallTime_99.9th_percentile" : 7620,
"TotalCallTime_TimeRangeCount_0-1" : 45,
"TotalCallTime_TimeRangeCount_3000-10000" : 1,
"exceptions.FailedSanityCheckException" : 0,
"exceptions.RegionMovedException" : 0,
"QueueCallTime_num_ops" : 46,
"QueueCallTime_min" : 0,
"QueueCallTime_max" : 130,
"QueueCallTime_mean" : 3,
"QueueCallTime_25th_percentile" : 32,
"QueueCallTime_median" : 65,
"QueueCallTime_75th_percentile" : 97,
"QueueCallTime_90th_percentile" : 117,
"QueueCallTime_95th_percentile" : 123,
"QueueCallTime_98th_percentile" : 127,
"QueueCallTime_99th_percentile" : 128,
"QueueCallTime_99.9th_percentile" : 129,
"QueueCallTime_TimeRangeCount_0-1" : 46,
"authenticationFailures" : 0,
"authorizationFailures" : 0,
"authenticationFallbacks" : 0,
"exceptions" : 0,
"RequestSize_num_ops" : 46,
"RequestSize_min" : 12,
"RequestSize_max" : 109,
"RequestSize_mean" : 41,
"RequestSize_25th_percentile" : 36,
"RequestSize_median" : 60,
"RequestSize_75th_percentile" : 84,
"RequestSize_90th_percentile" : 99,
"RequestSize_95th_percentile" : 104,
"RequestSize_98th_percentile" : 107,
"RequestSize_99th_percentile" : 108,
"RequestSize_99.9th_percentile" : 108,
"RequestSize_SizeRangeCount_0-10" : 46,
"ResponseSize_num_ops" : 46,
"ResponseSize_min" : 2,
"ResponseSize_max" : 485,
"ResponseSize_mean" : 18,
"ResponseSize_25th_percentile" : 122,
"ResponseSize_median" : 243,
"ResponseSize_75th_percentile" : 364,
"ResponseSize_90th_percentile" : 436,
"ResponseSize_95th_percentile" : 460,
"ResponseSize_98th_percentile" : 475,
"ResponseSize_99th_percentile" : 480,
"ResponseSize_99.9th_percentile" : 484,
"ResponseSize_SizeRangeCount_0-10" : 46,
"authenticationSuccesses" : 0,
"authorizationSuccesses" : 8,
"ProcessCallTime_num_ops" : 46,
"ProcessCallTime_min" : 0,
"ProcessCallTime_max" : 7518,
"ProcessCallTime_mean" : 183,
"ProcessCallTime_25th_percentile" : 300,
"ProcessCallTime_median" : 600,
"ProcessCallTime_75th_percentile" : 900,
"ProcessCallTime_90th_percentile" : 1081,
"ProcessCallTime_95th_percentile" : 1141,
"ProcessCallTime_98th_percentile" : 7089,
"ProcessCallTime_99th_percentile" : 7303,
"ProcessCallTime_99.9th_percentile" : 7496,
"ProcessCallTime_TimeRangeCount_0-1" : 45,
"ProcessCallTime_TimeRangeCount_3000-10000" : 1,
"exceptions.NotServingRegionException" : 0,
"sentBytes" : 4320,
"exceptions.RegionTooBusyException" : 0,
"receivedBytes" : 5141,
"exceptions.OutOfOrderScannerNextException" : 0,
"exceptions.multiResponseTooLarge" : 0,
"exceptions.UnknownScannerException" : 0
} ],
"beans" : [ {
"name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication",
"modelerType" : "RegionServer,sub=Replication",
"tag.Context" : "regionserver",
"tag.Hostname" : "quickstart.cloudera",
"sink.appliedOps" : 0,
"sink.appliedBatches" : 0,
"sink.ageOfLastAppliedOp" : 0
} ],
"beans" : [ {
"name" : "Hadoop:service=HBase,name=RegionServer,sub=Server",
"modelerType" : "RegionServer,sub=Server",
"tag.zookeeperQuorum" : "localhost:2181",
"tag.serverName" : "quickstart.cloudera,60020,1496810964011",
"tag.clusterId" : "ae292c1c-713f-4dbe-8a41-7c6e94a1a0e6",
"tag.Context" : "regionserver",
"tag.Hostname" : "quickstart.cloudera",
"regionCount" : 2,
"storeCount" : 2,
"hlogFileCount" : 2,
"hlogFileSize" : 0,
"storeFileCount" : 2,
"memStoreSize" : 864,
"storeFileSize" : 2461,
"regionServerStartTime" : 1496810964011,
"totalRequestCount" : 47,
"readRequestCount" : 14,
"writeRequestCount" : 4,
"checkMutateFailedCount" : 0,
"checkMutatePassedCount" : 0,
"storeFileIndexSize" : 856,
"staticIndexSize" : 140,
"staticBloomSize" : 4,
"mutationsWithoutWALCount" : 0,
"mutationsWithoutWALSize" : 0,
"percentFilesLocal" : 100.0,
"percentFilesLocalSecondaryRegions" : 0.0,
"splitQueueLength" : 0,
"compactionQueueLength" : 0,
"flushQueueLength" : 0,
"blockCacheFreeSize" : 500860056,
"blockCacheCount" : 1,
"blockCacheSize" : 516552,
"blockCacheHitCount" : 4,
"blockCacheHitCountPrimary" : 4,
"blockCacheMissCount" : 1,
"blockCacheMissCountPrimary" : 1,
"blockCacheEvictionCount" : 0,
"blockCacheEvictionCountPrimary" : 0,
"blockCacheCountHitPercent" : 80.0000011920929,
"blockCountHitPercent" : 80,
"blockCacheExpres**bleep**Percent" : 80.0000011920929,
"blockCacheFailedInsertionCount" : 0,
"updatesBlockedTime" : 0,
"flushedCellsCount" : 6,
"compactedCellsCount" : 0,
"majorCompactedCellsCount" : 0,
"flushedCellsSize" : 1344,
"compactedCellsSize" : 0,
"majorCompactedCellsSize" : 0,
"blockedRequestCount" : 0,
"hedgedReads" : 0,
"hedgedReadWins" : 0,
"mobCompactedFromMobCellsCount" : 0,
"mobCompactedIntoMobCellsCount" : 0,
"mobCompactedFromMobCellsSize" : 0,
"mobCompactedIntoMobCellsSize" : 0,
"mobFlushCount" : 0,
"mobFlushedCellsCount" : 0,
"mobFlushedCellsSize" : 0,
"mobScanCellsCount" : 0,
"mobScanCellsSize" : 0,
"mobFileCacheCount" : 0,
"mobFileCacheAccessCount" : 0,
"mobFileCacheMissCount" : 0,
"mobFileCacheEvictedCount" : 0,
"mobFileCacheHitPercent" : 0,
"PauseTimeWithGc_num_ops" : 0,
"PauseTimeWithGc_min" : 0,
"PauseTimeWithGc_max" : 0,
"PauseTimeWithGc_mean" : 0,
"PauseTimeWithGc_25th_percentile" : 0,
"PauseTimeWithGc_median" : 0,
"PauseTimeWithGc_75th_percentile" : 0,
"PauseTimeWithGc_90th_percentile" : 0,
"PauseTimeWithGc_95th_percentile" : 0,
"PauseTimeWithGc_98th_percentile" : 0,
"PauseTimeWithGc_99th_percentile" : 0,
"PauseTimeWithGc_99.9th_percentile" : 0,
"pauseWarnThresholdExceeded" : 0,
"splitSuccessCount" : 0,
"splitRequestCount" : 0,
"Append_num_ops" : 0,
"Append_min" : 0,
"Append_max" : 0,
"Append_mean" : 0,
"Append_25th_percentile" : 0,
"Append_median" : 0,
"Append_75th_percentile" : 0,
"Append_90th_percentile" : 0,
"Append_95th_percentile" : 0,
"Append_98th_percentile" : 0,
"Append_99th_percentile" : 0,
"Append_99.9th_percentile" : 0,
"Delete_num_ops" : 0,
"Delete_min" : 0,
"Delete_max" : 0,
"Delete_mean" : 0,
"Delete_25th_percentile" : 0,
"Delete_median" : 0,
"Delete_75th_percentile" : 0,
"Delete_90th_percentile" : 0,
"Delete_95th_percentile" : 0,
"Delete_98th_percentile" : 0,
"Delete_99th_percentile" : 0,
"Delete_99.9th_percentile" : 0,
"Mutate_num_ops" : 4,
"Mutate_min" : 4,
"Mutate_max" : 56,
"Mutate_mean" : 18,
"Mutate_25th_percentile" : 17,
"Mutate_median" : 30,
"Mutate_75th_percentile" : 43,
"Mutate_90th_percentile" : 50,
"Mutate_95th_percentile" : 53,
"Mutate_98th_percentile" : 54,
"Mutate_99th_percentile" : 55,
"Mutate_99.9th_percentile" : 55,
"Mutate_TimeRangeCount_0-1" : 4,
"ScanNext_num_ops" : 12,
"ScanNext_min" : 0,
"ScanNext_max" : 744,
"ScanNext_mean" : 390,
"ScanNext_25th_percentile" : 186,
"ScanNext_median" : 372,
"ScanNext_75th_percentile" : 558,
"ScanNext_90th_percentile" : 669,
"ScanNext_95th_percentile" : 706,
"ScanNext_98th_percentile" : 729,
"ScanNext_99th_percentile" : 736,
"ScanNext_99.9th_percentile" : 743,
"ScanNext_TimeRangeCount_0-1" : 12,
"slowDeleteCount" : 0,
"slowIncrementCount" : 0,
"FlushTime_num_ops" : 2,
"FlushTime_min" : 9574,
"FlushTime_max" : 193759,
"FlushTime_mean" : 101666,
"FlushTime_25th_percentile" : 10076,
"FlushTime_median" : 10578,
"FlushTime_75th_percentile" : 193260,
"FlushTime_90th_percentile" : 193559,
"FlushTime_95th_percentile" : 193659,
"FlushTime_98th_percentile" : 193719,
"FlushTime_99th_percentile" : 193739,
"FlushTime_99.9th_percentile" : 193757,
"FlushTime_TimeRangeCount_3000-10000" : 1,
"FlushTime_TimeRangeCount_120000-300000" : 1,
"Get_num_ops" : 5,
"Get_min" : 0,
"Get_max" : 55,
"Get_mean" : 11,
"Get_25th_percentile" : 13,
"Get_median" : 27,
"Get_75th_percentile" : 41,
"Get_90th_percentile" : 49,
"Get_95th_percentile" : 52,
"Get_98th_percentile" : 53,
"Get_99th_percentile" : 54,
"Get_99.9th_percentile" : 54,
"Get_TimeRangeCount_0-1" : 5,
"Replay_num_ops" : 0,
"Replay_min" : 0,
"Replay_max" : 0,
"Replay_mean" : 0,
"Replay_25th_percentile" : 0,
"Replay_median" : 0,
"Replay_75th_percentile" : 0,
"Replay_90th_percentile" : 0,
"Replay_95th_percentile" : 0,
"Replay_98th_percentile" : 0,
"Replay_99th_percentile" : 0,
"Replay_99.9th_percentile" : 0,
"slowGetCount" : 0,
"slowAppendCount" : 0,
"slowPutCount" : 0,
"PauseTimeWithoutGc_num_ops" : 0,
"PauseTimeWithoutGc_min" : 0,
"PauseTimeWithoutGc_max" : 0,
"PauseTimeWithoutGc_mean" : 0,
"PauseTimeWithoutGc_25th_percentile" : 0,
"PauseTimeWithoutGc_median" : 0,
"PauseTimeWithoutGc_75th_percentile" : 0,
"PauseTimeWithoutGc_90th_percentile" : 0,
"PauseTimeWithoutGc_95th_percentile" : 0,
"PauseTimeWithoutGc_98th_percentile" : 0,
"PauseTimeWithoutGc_99th_percentile" : 0,
"PauseTimeWithoutGc_99.9th_percentile" : 0,
"SplitTime_num_ops" : 0,
"SplitTime_min" : 0,
"SplitTime_max" : 0,
"SplitTime_mean" : 0,
"SplitTime_25th_percentile" : 0,
"SplitTime_median" : 0,
"SplitTime_75th_percentile" : 0,
"SplitTime_90th_percentile" : 0,
"SplitTime_95th_percentile" : 0,
"SplitTime_98th_percentile" : 0,
"SplitTime_99th_percentile" : 0,
"SplitTime_99.9th_percentile" : 0,
"pauseInfoThresholdExceeded" : 0,
"Increment_num_ops" : 0,
"Increment_min" : 0,
"Increment_max" : 0,
"Increment_mean" : 0,
"Increment_25th_percentile" : 0,
"Increment_median" : 0,
"Increment_75th_percentile" : 0,
"Increment_90th_percentile" : 0,
"Increment_95th_percentile" : 0,
"Increment_98th_percentile" : 0,
"Increment_99th_percentile" : 0,
"Increment_99.9th_percentile" : 0
} ]
}
2017-06-06 22:22:01,801 INFO [main-EventThread] regionserver.HRegionServer: STOPPED: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase regionserver:60020-0x15c80e11a9b0004 received expired from ZooKeeper, aborting
2017-06-06 22:22:01,801 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2017-06-06 22:22:01,801 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Stopping infoServer
2017-06-06 22:22:01,805 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-06-06 22:22:01,806 INFO [SplitLogWorker-quickstart:60020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting.
2017-06-06 22:22:01,823 INFO [SplitLogWorker-quickstart:60020] regionserver.SplitLogWorker: SplitLogWorker quickstart.cloudera,60020,1496810964011 exiting
2017-06-06 22:22:01,827 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
2017-06-06 22:22:01,832 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore.
2017-06-06 22:22:01,833 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2017-06-06 22:22:01,834 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2017-06-06 22:22:01,832 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2017-06-06 22:22:01,832 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2017-06-06 22:22:01,837 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: aborting server quickstart.cloudera,60020,1496810964011
2017-06-06 22:22:01,838 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15c80e11a9b0005
2017-06-06 22:22:01,843 INFO [StoreCloserThread-hbase:namespace,,1496811020881.6e8c9b0e20974475891f01654c517d9b.-1] regionserver.HStore: Closed info
2017-06-06 22:22:01,848 INFO [RS_CLOSE_REGION-quickstart:60020-0] regionserver.HRegion: Closed hbase:namespace,,1496811020881.6e8c9b0e20974475891f01654c517d9b.
2017-06-06 22:22:01,910 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-06-06 22:22:01,911 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51081, server: quickstart.cloudera/127.0.0.1:2181
2017-06-06 22:22:02,014 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZooKeeper: Session: 0x15c80e11a9b0005 closed
2017-06-06 22:22:02,015 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-06-06 22:22:02,015 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2017-06-06 22:22:02,015 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2017-06-06 22:22:02,015 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2017-06-06 22:22:02,016 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2017-06-06 22:22:02,023 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Waiting on 1 regions to close
2017-06-06 22:22:02,031 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore: Closed info
2017-06-06 22:22:02,033 INFO [RS_CLOSE_META-quickstart:60020-0] regionserver.HRegion: Closed hbase:meta,,1.1588230740
2017-06-06 22:22:02,224 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: stopping server quickstart.cloudera,60020,1496810964011; all regions closed.
2017-06-06 22:22:02,225 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing...
java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
2017-06-06 22:22:02,227 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing...
java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721)
2017-06-06 22:22:02,230 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Shutdown / close of WAL failed: java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting...
2017-06-06 22:22:02,242 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020 closing leases
2017-06-06 22:22:02,242 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020 closed leases
2017-06-06 22:22:02,243 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] hbase.ChoreService: Chore service for: quickstart.cloudera,60020,1496810964011 had [[ScheduledChore: Name: quickstart.cloudera,60020,1496810964011-MemstoreFlusherChore Period: 10000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region quickstart.cloudera,60020,1496810964011 Period: 120000 Unit: MILLISECONDS]] on shutdown
2017-06-06 22:22:06,656 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020.logRoller] regionserver.LogRoller: LogRoller exiting.
2017-06-06 22:22:06,722 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker closing leases
2017-06-06 22:22:06,723 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker closed leases
2017-06-06 22:22:07,883 INFO [RS_OPEN_META-quickstart:60020-0-MetaLogRoller] regionserver.LogRoller: LogRoller exiting.
2017-06-06 22:22:22,891 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts
2017-06-06 22:22:22,891 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZKUtil: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase Unable to list children of znode /hbase/replication/rs/quickstart.cloudera,60020,1496810964011
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/quickstart.cloudera,60020,1496810964011
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310)
at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2162)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088)
at java.lang.Thread.run(Thread.java:745)
2017-06-06 22:22:22,894 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZooKeeperWatcher: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/quickstart.cloudera,60020,1496810964011
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280)
at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310)
at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2162)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088)
at java.lang.Thread.run(Thread.java:745)
2017-06-06 22:22:22,896 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] ipc.RpcServer: Stopping server on 60020
2017-06-06 22:22:22,904 INFO [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: stopping
2017-06-06 22:22:22,928 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2017-06-06 22:22:22,928 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2017-06-06 22:22:37,975 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2017-06-06 22:22:37,975 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/quickstart.cloudera,60020,1496810964011
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1236)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1225)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1427)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1096)
at java.lang.Thread.run(Thread.java:745)
2017-06-06 22:22:37,978 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: stopping server quickstart.cloudera,60020,1496810964011; zookeeper connection closed.
2017-06-06 22:22:37,978 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: regionserver/quickstart.cloudera/127.0.0.1:60020 exiting
2017-06-06 22:22:37,981 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2676)
2017-06-06 22:22:37,996 INFO [Thread-7] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@439b2c10
2017-06-06 22:22:37,996 INFO [Thread-7] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2017-06-06 22:22:37,998 INFO [Thread-7] regionserver.ShutdownHook: Shutdown hook finished.
... View more
Labels:
- Labels:
-
Quickstart VM