Created 06-07-2017 12:21 AM
When I run QuickStart VMS for VMWARE on 12, HBase RegionServer stop automatically, please help me!
Here are logs,
hbase-hbase-regionserver-quickstart.cloudera.log:
2017-06-06 22:21:36,326 WARN [ResponseProcessor for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742697_1873] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742697_1873 java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1075) 2017-06-06 22:21:37,355 INFO [quickstart.cloudera,60020,1496810964011_ChoreService_1] hbase.ScheduledChore: Chore: quickstart.cloudera,60020,1496810964011-MemstoreFlusherChore missed its start time 2017-06-06 22:21:37,356 INFO [quickstart.cloudera,60020,1496810964011_ChoreService_2] hbase.ScheduledChore: Chore: CompactionChecker missed its start time 2017-06-06 22:21:40,490 WARN [ResponseProcessor for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742698_1874] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1886283405-127.0.0.1-1491389868205:blk_1073742698_1874 java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2272) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:235) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:1075) 2017-06-06 22:21:59,870 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] util.Sleeper: We slept 81086ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2017-06-06 22:22:00,095 INFO [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 81657ms for sessionid 0x15c80e11a9b0004, closing socket connection and attempting reconnect 2017-06-06 22:22:00,099 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x15c80e11a9b0005, likely server has closed socket, closing socket connection and attempting reconnect 2017-06-06 22:22:01,682 INFO [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2017-06-06 22:22:01,682 INFO [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51079, server: quickstart.cloudera/127.0.0.1:2181 2017-06-06 22:22:01,689 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region server quickstart.cloudera,60020,1496810964011: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase regionserver:60020-0x15c80e11a9b0004 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:700) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:611) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2017-06-06 22:22:01,689 INFO [main-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x15c80e11a9b0004 has expired, closing socket connection 2017-06-06 22:22:01,690 FATAL [main-EventThread] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2017-06-06 22:22:01,776 INFO [main-EventThread] regionserver.HRegionServer: Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "NonHeapMemoryUsage" : { "committed" : 136773632, "init" : 136773632, "max" : 184549376, "used" : 45819504 }, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 76742656, "init" : 79339968, "max" : 1253441536, "used" : 39087448 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "quickstart.cloudera", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 1, "numActiveHandler" : 0, "TotalCallTime_num_ops" : 46, "TotalCallTime_min" : 0, "TotalCallTime_max" : 7648, "TotalCallTime_mean" : 186, "TotalCallTime_25th_percentile" : 300, "TotalCallTime_median" : 600, "TotalCallTime_75th_percentile" : 900, "TotalCallTime_90th_percentile" : 1081, "TotalCallTime_95th_percentile" : 1141, "TotalCallTime_98th_percentile" : 7099, "TotalCallTime_99th_percentile" : 7373, "TotalCallTime_99.9th_percentile" : 7620, "TotalCallTime_TimeRangeCount_0-1" : 45, "TotalCallTime_TimeRangeCount_3000-10000" : 1, "exceptions.FailedSanityCheckException" : 0, "exceptions.RegionMovedException" : 0, "QueueCallTime_num_ops" : 46, "QueueCallTime_min" : 0, "QueueCallTime_max" : 130, "QueueCallTime_mean" : 3, "QueueCallTime_25th_percentile" : 32, "QueueCallTime_median" : 65, "QueueCallTime_75th_percentile" : 97, "QueueCallTime_90th_percentile" : 117, "QueueCallTime_95th_percentile" : 123, "QueueCallTime_98th_percentile" : 127, "QueueCallTime_99th_percentile" : 128, "QueueCallTime_99.9th_percentile" : 129, "QueueCallTime_TimeRangeCount_0-1" : 46, "authenticationFailures" : 0, "authorizationFailures" : 0, "authenticationFallbacks" : 0, "exceptions" : 0, "RequestSize_num_ops" : 46, "RequestSize_min" : 12, "RequestSize_max" : 109, "RequestSize_mean" : 41, "RequestSize_25th_percentile" : 36, "RequestSize_median" : 60, "RequestSize_75th_percentile" : 84, "RequestSize_90th_percentile" : 99, "RequestSize_95th_percentile" : 104, "RequestSize_98th_percentile" : 107, "RequestSize_99th_percentile" : 108, "RequestSize_99.9th_percentile" : 108, "RequestSize_SizeRangeCount_0-10" : 46, "ResponseSize_num_ops" : 46, "ResponseSize_min" : 2, "ResponseSize_max" : 485, "ResponseSize_mean" : 18, "ResponseSize_25th_percentile" : 122, "ResponseSize_median" : 243, "ResponseSize_75th_percentile" : 364, "ResponseSize_90th_percentile" : 436, "ResponseSize_95th_percentile" : 460, "ResponseSize_98th_percentile" : 475, "ResponseSize_99th_percentile" : 480, "ResponseSize_99.9th_percentile" : 484, "ResponseSize_SizeRangeCount_0-10" : 46, "authenticationSuccesses" : 0, "authorizationSuccesses" : 8, "ProcessCallTime_num_ops" : 46, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 7518, "ProcessCallTime_mean" : 183, "ProcessCallTime_25th_percentile" : 300, "ProcessCallTime_median" : 600, "ProcessCallTime_75th_percentile" : 900, "ProcessCallTime_90th_percentile" : 1081, "ProcessCallTime_95th_percentile" : 1141, "ProcessCallTime_98th_percentile" : 7089, "ProcessCallTime_99th_percentile" : 7303, "ProcessCallTime_99.9th_percentile" : 7496, "ProcessCallTime_TimeRangeCount_0-1" : 45, "ProcessCallTime_TimeRangeCount_3000-10000" : 1, "exceptions.NotServingRegionException" : 0, "sentBytes" : 4320, "exceptions.RegionTooBusyException" : 0, "receivedBytes" : 5141, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions.multiResponseTooLarge" : 0, "exceptions.UnknownScannerException" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "quickstart.cloudera", "sink.appliedOps" : 0, "sink.appliedBatches" : 0, "sink.ageOfLastAppliedOp" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:2181", "tag.serverName" : "quickstart.cloudera,60020,1496810964011", "tag.clusterId" : "ae292c1c-713f-4dbe-8a41-7c6e94a1a0e6", "tag.Context" : "regionserver", "tag.Hostname" : "quickstart.cloudera", "regionCount" : 2, "storeCount" : 2, "hlogFileCount" : 2, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 864, "storeFileSize" : 2461, "regionServerStartTime" : 1496810964011, "totalRequestCount" : 47, "readRequestCount" : 14, "writeRequestCount" : 4, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 856, "staticIndexSize" : 140, "staticBloomSize" : 4, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 100.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 500860056, "blockCacheCount" : 1, "blockCacheSize" : 516552, "blockCacheHitCount" : 4, "blockCacheHitCountPrimary" : 4, "blockCacheMissCount" : 1, "blockCacheMissCountPrimary" : 1, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 80.0000011920929, "blockCountHitPercent" : 80, "blockCacheExpres**bleep**Percent" : 80.0000011920929, "blockCacheFailedInsertionCount" : 0, "updatesBlockedTime" : 0, "flushedCellsCount" : 6, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 1344, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "blockedRequestCount" : 0, "hedgedReads" : 0, "hedgedReadWins" : 0, "mobCompactedFromMobCellsCount" : 0, "mobCompactedIntoMobCellsCount" : 0, "mobCompactedFromMobCellsSize" : 0, "mobCompactedIntoMobCellsSize" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "pauseWarnThresholdExceeded" : 0, "splitSuccessCount" : 0, "splitRequestCount" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Mutate_num_ops" : 4, "Mutate_min" : 4, "Mutate_max" : 56, "Mutate_mean" : 18, "Mutate_25th_percentile" : 17, "Mutate_median" : 30, "Mutate_75th_percentile" : 43, "Mutate_90th_percentile" : 50, "Mutate_95th_percentile" : 53, "Mutate_98th_percentile" : 54, "Mutate_99th_percentile" : 55, "Mutate_99.9th_percentile" : 55, "Mutate_TimeRangeCount_0-1" : 4, "ScanNext_num_ops" : 12, "ScanNext_min" : 0, "ScanNext_max" : 744, "ScanNext_mean" : 390, "ScanNext_25th_percentile" : 186, "ScanNext_median" : 372, "ScanNext_75th_percentile" : 558, "ScanNext_90th_percentile" : 669, "ScanNext_95th_percentile" : 706, "ScanNext_98th_percentile" : 729, "ScanNext_99th_percentile" : 736, "ScanNext_99.9th_percentile" : 743, "ScanNext_TimeRangeCount_0-1" : 12, "slowDeleteCount" : 0, "slowIncrementCount" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 9574, "FlushTime_max" : 193759, "FlushTime_mean" : 101666, "FlushTime_25th_percentile" : 10076, "FlushTime_median" : 10578, "FlushTime_75th_percentile" : 193260, "FlushTime_90th_percentile" : 193559, "FlushTime_95th_percentile" : 193659, "FlushTime_98th_percentile" : 193719, "FlushTime_99th_percentile" : 193739, "FlushTime_99.9th_percentile" : 193757, "FlushTime_TimeRangeCount_3000-10000" : 1, "FlushTime_TimeRangeCount_120000-300000" : 1, "Get_num_ops" : 5, "Get_min" : 0, "Get_max" : 55, "Get_mean" : 11, "Get_25th_percentile" : 13, "Get_median" : 27, "Get_75th_percentile" : 41, "Get_90th_percentile" : 49, "Get_95th_percentile" : 52, "Get_98th_percentile" : 53, "Get_99th_percentile" : 54, "Get_99.9th_percentile" : 54, "Get_TimeRangeCount_0-1" : 5, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "slowGetCount" : 0, "slowAppendCount" : 0, "slowPutCount" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0 } ] } 2017-06-06 22:22:01,801 INFO [main-EventThread] regionserver.HRegionServer: STOPPED: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase regionserver:60020-0x15c80e11a9b0004 received expired from ZooKeeper, aborting 2017-06-06 22:22:01,801 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread 2017-06-06 22:22:01,801 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Stopping infoServer 2017-06-06 22:22:01,805 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2017-06-06 22:22:01,806 INFO [SplitLogWorker-quickstart:60020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting. 2017-06-06 22:22:01,823 INFO [SplitLogWorker-quickstart:60020] regionserver.SplitLogWorker: SplitLogWorker quickstart.cloudera,60020,1496810964011 exiting 2017-06-06 22:22:01,827 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030 2017-06-06 22:22:01,832 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore. 2017-06-06 22:22:01,833 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly. 2017-06-06 22:22:01,834 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly. 2017-06-06 22:22:01,832 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting 2017-06-06 22:22:01,832 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting 2017-06-06 22:22:01,837 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: aborting server quickstart.cloudera,60020,1496810964011 2017-06-06 22:22:01,838 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15c80e11a9b0005 2017-06-06 22:22:01,843 INFO [StoreCloserThread-hbase:namespace,,1496811020881.6e8c9b0e20974475891f01654c517d9b.-1] regionserver.HStore: Closed info 2017-06-06 22:22:01,848 INFO [RS_CLOSE_REGION-quickstart:60020-0] regionserver.HRegion: Closed hbase:namespace,,1496811020881.6e8c9b0e20974475891f01654c517d9b. 2017-06-06 22:22:01,910 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2017-06-06 22:22:01,911 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020-SendThread(quickstart.cloudera:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:51081, server: quickstart.cloudera/127.0.0.1:2181 2017-06-06 22:22:02,014 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZooKeeper: Session: 0x15c80e11a9b0005 closed 2017-06-06 22:22:02,015 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020-EventThread] zookeeper.ClientCnxn: EventThread shut down 2017-06-06 22:22:02,015 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Split Thread to finish... 2017-06-06 22:22:02,015 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish... 2017-06-06 22:22:02,015 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish... 2017-06-06 22:22:02,016 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish... 2017-06-06 22:22:02,023 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Waiting on 1 regions to close 2017-06-06 22:22:02,031 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore: Closed info 2017-06-06 22:22:02,033 INFO [RS_CLOSE_META-quickstart:60020-0] regionserver.HRegion: Closed hbase:meta,,1.1588230740 2017-06-06 22:22:02,224 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: stopping server quickstart.cloudera,60020,1496810964011; all regions closed. 2017-06-06 22:22:02,225 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721) 2017-06-06 22:22:02,227 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing... java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1465) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1236) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:721) 2017-06-06 22:22:02,230 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Shutdown / close of WAL failed: java.io.IOException: All datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-19bc9d86-06e8-4b24-a09e-662206edaf90,DISK] are bad. Aborting... 2017-06-06 22:22:02,242 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020 closing leases 2017-06-06 22:22:02,242 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020 closed leases 2017-06-06 22:22:02,243 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] hbase.ChoreService: Chore service for: quickstart.cloudera,60020,1496810964011 had [[ScheduledChore: Name: quickstart.cloudera,60020,1496810964011-MemstoreFlusherChore Period: 10000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region quickstart.cloudera,60020,1496810964011 Period: 120000 Unit: MILLISECONDS]] on shutdown 2017-06-06 22:22:06,656 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020.logRoller] regionserver.LogRoller: LogRoller exiting. 2017-06-06 22:22:06,722 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker closing leases 2017-06-06 22:22:06,723 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker] regionserver.Leases: regionserver/quickstart.cloudera/127.0.0.1:60020.leaseChecker closed leases 2017-06-06 22:22:07,883 INFO [RS_OPEN_META-quickstart:60020-0-MetaLogRoller] regionserver.LogRoller: LogRoller exiting. 2017-06-06 22:22:22,891 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts 2017-06-06 22:22:22,891 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZKUtil: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase Unable to list children of znode /hbase/replication/rs/quickstart.cloudera,60020,1496810964011 org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/quickstart.cloudera,60020,1496810964011 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280) at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310) at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180) at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172) at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2162) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088) at java.lang.Thread.run(Thread.java:745) 2017-06-06 22:22:22,894 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.ZooKeeperWatcher: regionserver:60020-0x15c80e11a9b0004, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/replication/rs/quickstart.cloudera,60020,1496810964011 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:456) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:484) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1476) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1398) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1280) at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:310) at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180) at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172) at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2162) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1088) at java.lang.Thread.run(Thread.java:745) 2017-06-06 22:22:22,896 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] ipc.RpcServer: Stopping server on 60020 2017-06-06 22:22:22,904 INFO [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: stopping 2017-06-06 22:22:22,928 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped 2017-06-06 22:22:22,928 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping 2017-06-06 22:22:37,975 ERROR [regionserver/quickstart.cloudera/127.0.0.1:60020] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts 2017-06-06 22:22:37,975 WARN [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/quickstart.cloudera,60020,1496810964011 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1236) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1225) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1427) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1096) at java.lang.Thread.run(Thread.java:745) 2017-06-06 22:22:37,978 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: stopping server quickstart.cloudera,60020,1496810964011; zookeeper connection closed. 2017-06-06 22:22:37,978 INFO [regionserver/quickstart.cloudera/127.0.0.1:60020] regionserver.HRegionServer: regionserver/quickstart.cloudera/127.0.0.1:60020 exiting 2017-06-06 22:22:37,981 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting java.lang.RuntimeException: HRegionServer Aborted at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2676) 2017-06-06 22:22:37,996 INFO [Thread-7] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@439b2c10 2017-06-06 22:22:37,996 INFO [Thread-7] regionserver.ShutdownHook: Starting fs shutdown hook thread. 2017-06-06 22:22:37,998 INFO [Thread-7] regionserver.ShutdownHook: Shutdown hook finished.
Created 11-08-2017 12:07 AM
Created 11-12-2021 01:30 AM
did it finally resolved?
Created 11-13-2021 06:51 AM
Hello @xgxshtc
We observed you have posted the concerned ask in a New Post [1] as the concerned Post is ~4Years Old. While the Current Post is Unresolved, We shall wait on your Team's review on [1] before confirming the Solution on the Current Post as well.
Regards, Smarak