2017-06-13 19:26:57,813 INFO  [sync.4] wal.FSHLog: Slow sync cost: 176 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-6e0a3279-95d2-4bff-ac72-ed8e01ad47ab,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-3109a88b-277e-4d45-8cfb-8a5f3bf57adb,DISK]]
2017-06-13 19:27:16,010 INFO  [sync.0] wal.FSHLog: Slow sync cost: 3357 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-6e0a3279-95d2-4bff-ac72-ed8e01ad47ab,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-3109a88b-277e-4d45-8cfb-8a5f3bf57adb,DISK]]
2017-06-13 19:29:52,286 INFO  [sync.2] wal.FSHLog: Slow sync cost: 814 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-6e0a3279-95d2-4bff-ac72-ed8e01ad47ab,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-3109a88b-277e-4d45-8cfb-8a5f3bf57adb,DISK]]
2017-06-13 19:29:52,320 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497381777863 with entries=459, filesize=121.72 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497382192286
2017-06-13 19:29:56,543 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.24 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:29:57,497 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36713918, memsize=128.2 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/77b91ba80c3b428c8137d4fdd7d8c4db
2017-06-13 19:29:57,511 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/77b91ba80c3b428c8137d4fdd7d8c4db, entries=297093, sequenceid=36713918, filesize=13.5 M
2017-06-13 19:29:57,514 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.24 MB/134474536, currentsize=9.45 MB/9911168 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 971ms, sequenceid=36713918, compaction requested=true
2017-06-13 19:29:57,516 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 19:29:57,516 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 19:29:57,517 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059810864, freeSize=213686992, maxSize=4273497856, heapSize=4059810864, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 19:29:59,349 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 0459e8bcdd514367850cf5f1b6865cb5(size=13.5 M), total size for store is 188.4 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 19:29:59,349 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21466959833962301; duration=1sec
2017-06-13 19:30:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.79 MB, max=3.98 GB, blockCount=61084, accesses=203806427, hits=104274954, hitRatio=51.16%, , cachingAccesses=200484917, cachingHits=104249453, cachingHitsRatio=52.00%, evictions=41691, evicted=94389104, evictedPerRun=2264.016357421875
2017-06-13 19:31:40,483 INFO  [sync.3] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-3109a88b-277e-4d45-8cfb-8a5f3bf57adb,DISK]]
2017-06-13 19:33:29,814 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.35 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:33:30,700 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36714436, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/cb78276b6f1a4a3b83f983b9af467af3
2017-06-13 19:33:30,715 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/cb78276b6f1a4a3b83f983b9af467af3, entries=297093, sequenceid=36714436, filesize=13.5 M
2017-06-13 19:33:30,716 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.35 MB/134579904, currentsize=11.22 MB/11769760 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 902ms, sequenceid=36714436, compaction requested=true
2017-06-13 19:33:31,732 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497382192286 with entries=480, filesize=122.01 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497382411683
2017-06-13 19:35:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61084, accesses=203825169, hits=104291644, hitRatio=51.17%, , cachingAccesses=200501608, cachingHits=104266142, cachingHitsRatio=52.00%, evictions=41721, evicted=94389106, evictedPerRun=2262.388427734375
2017-06-13 19:35:25,453 INFO  [sync.4] wal.FSHLog: Slow sync cost: 763 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 19:35:29,415 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.37 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:35:30,340 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36714999, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/77ad6506a5a14f36806db7e05a2191be
2017-06-13 19:35:30,352 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/77ad6506a5a14f36806db7e05a2191be, entries=297093, sequenceid=36714999, filesize=13.5 M
2017-06-13 19:35:30,353 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.37 MB/134607424, currentsize=8.37 MB/8781352 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 938ms, sequenceid=36714999, compaction requested=true
2017-06-13 19:35:30,355 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 19:35:30,355 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 19:35:30,356 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 19:35:32,333 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 0c2fe787017149dbb80759224fbcd288(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 19:35:32,333 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21467292672852170; duration=1sec
2017-06-13 19:35:49,165 INFO  [sync.4] wal.FSHLog: Slow sync cost: 6034 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 19:35:55,268 INFO  [sync.1] wal.FSHLog: Slow sync cost: 5976 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 19:36:03,431 INFO  [sync.0] wal.FSHLog: Slow sync cost: 6166 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 19:37:05,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. after a delay of 12284
2017-06-13 19:37:15,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. after a delay of 8898
2017-06-13 19:37:17,874 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c., current region memstore size 35.85 KB, and 2/2 column families' memstores are being flushed.
2017-06-13 19:37:17,916 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11044, memsize=16.6 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp/88304473009a4397975684304bf45245
2017-06-13 19:37:17,937 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11044, memsize=19.3 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp/a0349e5d426841f994385915d2035689
2017-06-13 19:37:17,948 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/ch/88304473009a4397975684304bf45245, entries=85, sequenceid=11044, filesize=5.8 K
2017-06-13 19:37:17,955 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/info/a0349e5d426841f994385915d2035689, entries=96, sequenceid=11044, filesize=6.5 K
2017-06-13 19:37:17,957 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~35.85 KB/36712, currentsize=0 B/0 for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. in 83ms, sequenceid=11044, compaction requested=true
2017-06-13 19:37:17,958 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on ch in region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c.
2017-06-13 19:37:17,958 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in ch of SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp, totalSize=70.2 K
2017-06-13 19:37:17,959 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 19:37:18,035 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 (all) file(s) in ch of SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. into 6cf476b15f3b44a2bf2acadba9ba0e40(size=59.8 K), total size for store is 59.8 K. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 19:37:18,035 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c., storeName=ch, fileCount=3, fileSize=70.2 K, priority=7, time=21467400275697404; duration=0sec
2017-06-13 19:37:18,037 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c.
2017-06-13 19:37:18,037 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp, totalSize=106.9 K
2017-06-13 19:37:18,038 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 19:37:18,106 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 (all) file(s) in info of SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. into 9c6f90b8826342fb92964e5a28cbfa9b(size=95.8 K), total size for store is 95.8 K. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 19:37:18,106 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c., storeName=info, fileCount=3, fileSize=106.9 K, priority=7, time=21467400354472990; duration=0sec
2017-06-13 19:39:34,085 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497382411683 with entries=476, filesize=121.88 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497382774031
2017-06-13 19:39:34,088 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497378997780 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497378997780
2017-06-13 19:39:34,103 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497379424559 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497379424559
2017-06-13 19:39:34,117 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497380561261 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497380561261
2017-06-13 19:39:34,139 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497380846747 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497380846747
2017-06-13 19:39:42,711 INFO  [sync.0] wal.FSHLog: Slow sync cost: 248 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 19:39:43,560 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.33 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:39:44,488 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36715521, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/25c7b96b45dd40838621c8b000020b6c
2017-06-13 19:39:44,525 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/25c7b96b45dd40838621c8b000020b6c, entries=297093, sequenceid=36715521, filesize=13.5 M
2017-06-13 19:39:44,537 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.33 MB/134565472, currentsize=7.52 MB/7882040 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 977ms, sequenceid=36715521, compaction requested=true
2017-06-13 19:40:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61084, accesses=203827399, hits=104291659, hitRatio=51.17%, , cachingAccesses=200501608, cachingHits=104266142, cachingHitsRatio=52.00%, evictions=41751, evicted=94389106, evictedPerRun=2260.7626953125
2017-06-13 19:41:40,203 INFO  [sync.1] wal.FSHLog: Slow sync cost: 848 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 19:41:41,153 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.19 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:41:42,094 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36716072, memsize=128.2 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/b640653546b646f3950722b95d559e5e
2017-06-13 19:41:42,110 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/b640653546b646f3950722b95d559e5e, entries=297093, sequenceid=36716072, filesize=13.5 M
2017-06-13 19:41:42,111 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.19 MB/134421224, currentsize=6.32 MB/6627544 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 959ms, sequenceid=36716072, compaction requested=true
2017-06-13 19:41:42,113 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 19:41:42,113 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 19:41:42,114 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 19:41:43,956 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into a65a3d7bc5cc43c4a801d679490a198e(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 19:41:43,956 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21467664430647038; duration=1sec
2017-06-13 19:42:07,725 INFO  [sync.1] wal.FSHLog: Slow sync cost: 3664 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 19:43:38,255 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497382774031 with entries=468, filesize=121.98 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383018217
2017-06-13 19:45:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61084, accesses=203829612, hits=104291659, hitRatio=51.17%, , cachingAccesses=200501608, cachingHits=104266142, cachingHitsRatio=52.00%, evictions=41781, evicted=94389106, evictedPerRun=2259.139404296875
2017-06-13 19:45:25,953 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.12 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:45:26,763 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36716604, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/7883d4b7cc494fc0a4b4bd7f3288b19d
2017-06-13 19:45:26,778 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/7883d4b7cc494fc0a4b4bd7f3288b19d, entries=297093, sequenceid=36716604, filesize=13.5 M
2017-06-13 19:45:26,779 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.12 MB/134339448, currentsize=13.43 MB/14082824 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 826ms, sequenceid=36716604, compaction requested=true
2017-06-13 19:47:19,412 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383018217 with entries=475, filesize=121.84 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383239374
2017-06-13 19:47:20,087 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.33 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:47:20,885 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36717145, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/714c9727acb34037aa57ab318f3e451e
2017-06-13 19:47:20,901 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/714c9727acb34037aa57ab318f3e451e, entries=297093, sequenceid=36717145, filesize=13.5 M
2017-06-13 19:47:20,902 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.33 MB/134567408, currentsize=783.78 KB/802592 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 815ms, sequenceid=36717145, compaction requested=true
2017-06-13 19:47:20,904 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 19:47:20,904 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 19:47:20,905 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 19:47:22,745 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 80b15ae5a2cb4a9d953805c012abc66d(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 19:47:22,745 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21468003221463445; duration=1sec
2017-06-13 19:49:35,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. after a delay of 5559
2017-06-13 19:49:41,148 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228., current region memstore size 48.35 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:49:41,468 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=266104, memsize=48.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/.tmp/8fe521d6651946829c46185f3ade7d55
2017-06-13 19:49:41,480 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/info/8fe521d6651946829c46185f3ade7d55, entries=245014, sequenceid=266104, filesize=2.7 M
2017-06-13 19:49:41,481 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~48.35 MB/50700840, currentsize=0 B/0 for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. in 333ms, sequenceid=266104, compaction requested=true
2017-06-13 19:50:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61084, accesses=203831825, hits=104291659, hitRatio=51.17%, , cachingAccesses=200501608, cachingHits=104266142, cachingHitsRatio=52.00%, evictions=41811, evicted=94389106, evictedPerRun=2257.5185546875
2017-06-13 19:51:00,715 INFO  [sync.4] wal.FSHLog: Slow sync cost: 782 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-8ab76f9c-ee05-4ec0-897a-8718ab89635f,DISK]]
2017-06-13 19:51:00,716 INFO  [sync.0] wal.FSHLog: Slow sync cost: 775 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-8ab76f9c-ee05-4ec0-897a-8718ab89635f,DISK]]
2017-06-13 19:51:03,301 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:51:04,179 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36717687, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/0c3ba648837643e2b449804be984edfc
2017-06-13 19:51:04,214 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/0c3ba648837643e2b449804be984edfc, entries=297093, sequenceid=36717687, filesize=13.5 M
2017-06-13 19:51:04,215 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232744, currentsize=11.78 MB/12354104 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 914ms, sequenceid=36717687, compaction requested=true
2017-06-13 19:51:06,372 INFO  [sync.1] wal.FSHLog: Slow sync cost: 472 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-8ab76f9c-ee05-4ec0-897a-8718ab89635f,DISK]]
2017-06-13 19:51:08,644 INFO  [sync.3] wal.FSHLog: Slow sync cost: 506 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-8ab76f9c-ee05-4ec0-897a-8718ab89635f,DISK]]
2017-06-13 19:52:50,613 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383239374 with entries=630, filesize=121.69 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383570583
2017-06-13 19:52:50,613 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497379677179 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497379677179
2017-06-13 19:54:38,983 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.30 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:54:39,842 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36718226, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/88939d8c42e046948f16ab7d4b3eb37a
2017-06-13 19:54:39,859 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/88939d8c42e046948f16ab7d4b3eb37a, entries=297093, sequenceid=36718226, filesize=13.5 M
2017-06-13 19:54:39,860 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.30 MB/134531800, currentsize=10.32 MB/10822400 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 877ms, sequenceid=36718226, compaction requested=true
2017-06-13 19:54:39,905 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 19:54:39,906 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.5 M
2017-06-13 19:54:39,906 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 19:54:41,779 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 4260122ca46b49ada439a0556c4b2707(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 19:54:41,779 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.5 M, priority=6, time=21468442223247946; duration=1sec
2017-06-13 19:55:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61084, accesses=203834038, hits=104291659, hitRatio=51.16%, , cachingAccesses=200501608, cachingHits=104266142, cachingHitsRatio=52.00%, evictions=41841, evicted=94389106, evictedPerRun=2255.89990234375
2017-06-13 19:56:31,710 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383570583 with entries=468, filesize=121.74 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383791680
2017-06-13 19:56:32,114 INFO  [sync.0] wal.FSHLog: Slow sync cost: 132 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-120b90e1-5d80-447d-b8db-b8ea53661e88,DISK]]
2017-06-13 19:56:32,114 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.05 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:56:34,164 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36718770, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/78eaaff499714264996fa418a1f9e8b0
2017-06-13 19:56:34,188 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/78eaaff499714264996fa418a1f9e8b0, entries=297093, sequenceid=36718770, filesize=13.5 M
2017-06-13 19:56:34,191 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134272664, currentsize=23.33 MB/24458904 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 2077ms, sequenceid=36718770, compaction requested=true
2017-06-13 19:56:44,106 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789., current region memstore size 128.04 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:56:44,483 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2081856, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/.tmp/8277265bcc0c493286ca0860842cc67d
2017-06-13 19:56:44,495 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/info/8277265bcc0c493286ca0860842cc67d, entries=37195, sequenceid=2081856, filesize=726.3 K
2017-06-13 19:56:44,497 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.04 MB/134264784, currentsize=423.94 KB/434112 for region ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. in 390ms, sequenceid=2081856, compaction requested=false
2017-06-13 19:57:05,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region WebAnalyticsUserFlow,cDGwVW7RALanguageEnglish01493633934,1497288233889.bd9dd4b2d0aec2181759670c9fc4ec2b. after a delay of 15454
2017-06-13 19:57:15,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region WebAnalyticsUserFlow,cDGwVW7RALanguageEnglish01493633934,1497288233889.bd9dd4b2d0aec2181759670c9fc4ec2b. after a delay of 5057
2017-06-13 19:57:21,043 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for WebAnalyticsUserFlow,cDGwVW7RALanguageEnglish01493633934,1497288233889.bd9dd4b2d0aec2181759670c9fc4ec2b., current region memstore size 94.17 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 19:57:21,656 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=263371, memsize=94.2 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/bd9dd4b2d0aec2181759670c9fc4ec2b/.tmp/b6833c8da8744f34be39d6338e1bd6bf
2017-06-13 19:57:21,669 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/bd9dd4b2d0aec2181759670c9fc4ec2b/info/b6833c8da8744f34be39d6338e1bd6bf, entries=484918, sequenceid=263371, filesize=4.8 M
2017-06-13 19:57:21,672 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~94.17 MB/98744904, currentsize=0 B/0 for region WebAnalyticsUserFlow,cDGwVW7RALanguageEnglish01493633934,1497288233889.bd9dd4b2d0aec2181759670c9fc4ec2b. in 629ms, sequenceid=263371, compaction requested=true
2017-06-13 20:00:09,878 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.41 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:00:11,233 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36719303, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/f5649fdc2a3c403f957c013ac2fc0d27
2017-06-13 20:00:11,258 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/f5649fdc2a3c403f957c013ac2fc0d27, entries=297093, sequenceid=36719303, filesize=13.5 M
2017-06-13 20:00:11,259 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.41 MB/134651736, currentsize=11.58 MB/12139856 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1381ms, sequenceid=36719303, compaction requested=true
2017-06-13 20:00:11,262 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:00:11,262 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.5 M
2017-06-13 20:00:11,262 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:00:13,137 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 06d6320d007f45d5bba52d4a5d1f8307(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 20:00:13,137 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.5 M, priority=6, time=21468773579574457; duration=1sec
2017-06-13 20:00:13,918 INFO  [sync.4] wal.FSHLog: Slow sync cost: 368 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-120b90e1-5d80-447d-b8db-b8ea53661e88,DISK]]
2017-06-13 20:00:16,329 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 1191 ms, current pipeline: []
2017-06-13 20:00:16,360 INFO  [sync.0] wal.FSHLog: Slow sync cost: 249 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 20:00:16,360 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383791680 with entries=692, filesize=128.09 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384015123
2017-06-13 20:00:16,361 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497380066217 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497380066217
2017-06-13 20:00:16,375 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497380226691 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497380226691
2017-06-13 20:00:16,388 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497382192286 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497382192286
2017-06-13 20:00:16,402 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383018217 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497383018217
2017-06-13 20:00:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61084, accesses=203836251, hits=104291659, hitRatio=51.16%, , cachingAccesses=200501608, cachingHits=104266142, cachingHitsRatio=52.00%, evictions=41871, evicted=94389106, evictedPerRun=2254.283447265625
2017-06-13 20:02:08,768 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.38 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:02:09,661 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36719855, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/5ad6a5e1be3d405abaabecf3aa63cba1
2017-06-13 20:02:09,673 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/5ad6a5e1be3d405abaabecf3aa63cba1, entries=297093, sequenceid=36719855, filesize=13.5 M
2017-06-13 20:02:09,674 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.38 MB/134614016, currentsize=12.55 MB/13156072 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 906ms, sequenceid=36719855, compaction requested=true
2017-06-13 20:04:13,830 INFO  [sync.3] wal.FSHLog: Slow sync cost: 5446 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 20:05:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61084, accesses=203837779, hits=104291667, hitRatio=51.16%, , cachingAccesses=200501616, cachingHits=104266150, cachingHitsRatio=52.00%, evictions=41901, evicted=94389106, evictedPerRun=2252.66943359375
2017-06-13 20:05:43,744 INFO  [RS_OPEN_META-aps-hadoop4:16020-0-MetaLogRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497380743514.meta with entries=0, filesize=91 B; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497384343708.meta
2017-06-13 20:05:43,744 INFO  [RS_OPEN_META-aps-hadoop4:16020-0-MetaLogRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497380743514.meta to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004..meta.1497380743514.meta
2017-06-13 20:06:09,794 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.34 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:06:10,486 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384015123 with entries=459, filesize=121.63 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384370428
2017-06-13 20:06:10,587 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36720378, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/dfccef1f3bfa47c390c88179b372d965
2017-06-13 20:06:10,605 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/dfccef1f3bfa47c390c88179b372d965, entries=297093, sequenceid=36720378, filesize=13.5 M
2017-06-13 20:06:10,606 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.34 MB/134574392, currentsize=6.75 MB/7076056 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 812ms, sequenceid=36720378, compaction requested=true
2017-06-13 20:06:10,608 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:06:10,608 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.5 M
2017-06-13 20:06:10,609 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:06:12,601 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into d16d9705fbf04f80bd4f548ba31a7af7(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 20:06:12,601 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.5 M, priority=6, time=21469132926022527; duration=1sec
2017-06-13 20:06:16,783 INFO  [sync.3] wal.FSHLog: Slow sync cost: 230 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 20:07:41,563 INFO  [sync.4] wal.FSHLog: Slow sync cost: 1988 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 20:08:11,134 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.02 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:08:12,043 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36720938, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/fd819888a2e849d98139dadad4a60540
2017-06-13 20:08:12,055 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/fd819888a2e849d98139dadad4a60540, entries=297093, sequenceid=36720938, filesize=13.5 M
2017-06-13 20:08:12,056 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.02 MB/134241048, currentsize=10.41 MB/10912024 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 922ms, sequenceid=36720938, compaction requested=true
2017-06-13 20:09:56,403 INFO  [sync.0] wal.FSHLog: Slow sync cost: 869 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 20:10:00,202 INFO  [sync.2] wal.FSHLog: Slow sync cost: 259 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 20:10:01,991 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384370428 with entries=495, filesize=121.96 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384601931
2017-06-13 20:10:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61084, accesses=203839992, hits=104291667, hitRatio=51.16%, , cachingAccesses=200501616, cachingHits=104266150, cachingHitsRatio=52.00%, evictions=41931, evicted=94389106, evictedPerRun=2251.057861328125
2017-06-13 20:11:49,680 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.07 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:11:50,552 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36721455, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/ce6d94d324424d7f90d483ed12d76e94
2017-06-13 20:11:50,620 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/ce6d94d324424d7f90d483ed12d76e94, entries=297093, sequenceid=36721455, filesize=13.5 M
2017-06-13 20:11:50,625 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134287448, currentsize=7.35 MB/7705336 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 945ms, sequenceid=36721455, compaction requested=true
2017-06-13 20:11:50,645 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:11:50,645 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 20:11:50,645 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:11:52,544 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 3354297002054de18b87484432202871(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 20:11:52,544 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21469472962571777; duration=1sec
2017-06-13 20:12:05,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. after a delay of 8382
2017-06-13 20:12:13,971 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf., current region memstore size 128.63 KB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:12:14,080 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=534353, memsize=128.6 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/.tmp/882d6b7ca6b546afb0f0f4f210e378d0
2017-06-13 20:12:14,101 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/info/882d6b7ca6b546afb0f0f4f210e378d0, entries=176, sequenceid=534353, filesize=8.5 K
2017-06-13 20:12:14,102 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.63 KB/131712, currentsize=0 B/0 for region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. in 131ms, sequenceid=534353, compaction requested=true
2017-06-13 20:12:14,103 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf.
2017-06-13 20:12:14,103 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/.tmp, totalSize=521.7 K
2017-06-13 20:12:14,105 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:12:14,475 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 (all) file(s) in info of CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. into 152cb7dd31934dffb04ca82c3a0a55d2(size=503.2 K), total size for store is 503.2 K. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 20:12:14,475 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf., storeName=info, fileCount=3, fileSize=521.7 K, priority=7, time=21469496420878521; duration=0sec
2017-06-13 20:13:55,175 INFO  [sync.2] wal.FSHLog: Slow sync cost: 341 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK]]
2017-06-13 20:14:00,908 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.24 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:14:01,991 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36722021, memsize=128.2 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/9a95f35f45fe49559ea726c7abfdb010
2017-06-13 20:14:02,009 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/9a95f35f45fe49559ea726c7abfdb010, entries=297093, sequenceid=36722021, filesize=13.5 M
2017-06-13 20:14:02,010 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.24 MB/134474536, currentsize=8.22 MB/8623776 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1102ms, sequenceid=36722021, compaction requested=true
2017-06-13 20:14:13,793 INFO  [sync.4] wal.FSHLog: Slow sync cost: 236 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK]]
2017-06-13 20:15:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61084, accesses=203842299, hits=104291670, hitRatio=51.16%, , cachingAccesses=200501619, cachingHits=104266153, cachingHitsRatio=52.00%, evictions=41961, evicted=94389106, evictedPerRun=2249.448486328125
2017-06-13 20:15:51,476 INFO  [sync.2] wal.FSHLog: Slow sync cost: 1682 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK]]
2017-06-13 20:15:52,220 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384601931 with entries=461, filesize=121.77 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384952168
2017-06-13 20:15:52,220 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497381102270 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497381102270
2017-06-13 20:15:52,240 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497381524059 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497381524059
2017-06-13 20:15:52,259 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497382411683 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497382411683
2017-06-13 20:15:52,279 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383570583 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497383570583
2017-06-13 20:15:55,673 INFO  [sync.1] wal.FSHLog: Slow sync cost: 986 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK]]
2017-06-13 20:17:15,610 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcwE41OAPE2K6,1489986614646.18690781354093c773e1a351396d0479.
2017-06-13 20:17:15,610 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcwE41OAPE2K6,1489986614646.18690781354093c773e1a351396d0479. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/18690781354093c773e1a351396d0479/.tmp, totalSize=1.6 G
2017-06-13 20:17:15,692 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61084, currentSize=4059750664, freeSize=213747192, maxSize=4273497856, heapSize=4059750664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:17:51,909 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.35 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:17:53,065 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36722538, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/fd219ef9d94e496f8d8b5001d682a87a
2017-06-13 20:17:53,087 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/fd219ef9d94e496f8d8b5001d682a87a, entries=297093, sequenceid=36722538, filesize=13.5 M
2017-06-13 20:17:53,088 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.35 MB/134579904, currentsize=15.01 MB/15734896 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1179ms, sequenceid=36722538, compaction requested=true
2017-06-13 20:17:56,852 INFO  [sync.4] wal.FSHLog: Slow sync cost: 268 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK]]
2017-06-13 20:19:57,838 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384952168 with entries=475, filesize=121.91 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497385197787
2017-06-13 20:19:59,988 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.37 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:20:00,838 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36723101, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/52cdd894d140470b9d7fa6b3bdcde998
2017-06-13 20:20:00,851 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/52cdd894d140470b9d7fa6b3bdcde998, entries=297093, sequenceid=36723101, filesize=13.5 M
2017-06-13 20:20:00,852 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.37 MB/134607424, currentsize=7.62 MB/7990904 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 864ms, sequenceid=36723101, compaction requested=true
2017-06-13 20:20:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61082, accesses=203956746, hits=104297004, hitRatio=51.14%, , cachingAccesses=200506953, cachingHits=104271480, cachingHitsRatio=52.00%, evictions=41991, evicted=94389115, evictedPerRun=2247.841552734375
2017-06-13 20:20:42,622 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed major compaction of 3 (all) file(s) in info of SMSCampaignStatus,zAcwE41OAPE2K6,1489986614646.18690781354093c773e1a351396d0479. into d06a019fd1ee40bb8744013ec8a677a2(size=1.5 G), total size for store is 1.5 G. This selection was in queue for 0sec, and took 3mins, 27sec to execute.
2017-06-13 20:20:42,623 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcwE41OAPE2K6,1489986614646.18690781354093c773e1a351396d0479., storeName=info, fileCount=3, fileSize=1.6 G, priority=7, time=21469797927428890; duration=3mins, 27sec
2017-06-13 20:20:42,625 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:20:42,625 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 4 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=53.9 M
2017-06-13 20:20:42,626 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61082, currentSize=4059749200, freeSize=213748656, maxSize=4273497856, heapSize=4059749200, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:20:45,231 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 4 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 099f2cf2098d46bc8cfaad8de8d0cd12(size=13.5 M), total size for store is 188.4 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 20:20:45,232 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=4, fileSize=53.9 M, priority=5, time=21470004942870565; duration=2sec
2017-06-13 20:23:53,046 INFO  [sync.0] wal.FSHLog: Slow sync cost: 230 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 20:23:53,406 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.33 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:23:54,192 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36723623, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/473970e725d0489886f66d33df648994
2017-06-13 20:23:54,206 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/473970e725d0489886f66d33df648994, entries=297093, sequenceid=36723623, filesize=13.5 M
2017-06-13 20:23:54,207 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.33 MB/134565472, currentsize=12.96 MB/13593840 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 801ms, sequenceid=36723623, compaction requested=true
2017-06-13 20:23:55,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. after a delay of 19782
2017-06-13 20:23:57,378 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497385197787 with entries=469, filesize=122.13 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497385437346
2017-06-13 20:24:05,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. after a delay of 15204
2017-06-13 20:24:15,372 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 9.84 KB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:24:15,595 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1694419, memsize=9.8 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/8ab48cd4d08e4a3885aa6e81f0436fc0
2017-06-13 20:24:15,610 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/8ab48cd4d08e4a3885aa6e81f0436fc0, entries=56, sequenceid=1694419, filesize=5.7 K
2017-06-13 20:24:15,611 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~9.84 KB/10072, currentsize=0 B/0 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 239ms, sequenceid=1694419, compaction requested=true
2017-06-13 20:25:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.85 MB, max=3.98 GB, blockCount=61082, accesses=203971571, hits=104297006, hitRatio=51.13%, , cachingAccesses=200506955, cachingHits=104271482, cachingHitsRatio=52.00%, evictions=42021, evicted=94389115, evictedPerRun=2246.236572265625
2017-06-13 20:25:48,076 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.19 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:25:48,949 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36724174, memsize=128.2 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/b4644ef59d7b476292382a98f221b5a2
2017-06-13 20:25:48,978 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/b4644ef59d7b476292382a98f221b5a2, entries=297093, sequenceid=36724174, filesize=13.5 M
2017-06-13 20:25:48,979 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.19 MB/134421224, currentsize=6.32 MB/6627544 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 903ms, sequenceid=36724174, compaction requested=true
2017-06-13 20:25:48,981 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:25:48,981 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 20:25:48,982 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61086, currentSize=4059769936, freeSize=213727920, maxSize=4273497856, heapSize=4059769936, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:25:50,887 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into d60059f09bd24e38a1b3e01a1955e8ac(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 20:25:50,887 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21470311298767699; duration=1sec
2017-06-13 20:29:31,046 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497385437346 with entries=469, filesize=122.00 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497385771002
2017-06-13 20:29:31,047 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497381777863 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497381777863
2017-06-13 20:29:31,746 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.12 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:29:34,081 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36724706, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/aa65c127f27048f091a215c3c0ec2af4
2017-06-13 20:29:34,096 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/aa65c127f27048f091a215c3c0ec2af4, entries=297093, sequenceid=36724706, filesize=13.5 M
2017-06-13 20:29:34,097 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.12 MB/134339448, currentsize=32.11 MB/33667032 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 2351ms, sequenceid=36724706, compaction requested=true
2017-06-13 20:29:45,497 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789., current region memstore size 128.25 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:29:45,847 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2082447, memsize=128.2 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/.tmp/f03f048505324bca810329c3afa2a12e
2017-06-13 20:29:45,861 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/info/f03f048505324bca810329c3afa2a12e, entries=37341, sequenceid=2082447, filesize=726.7 K
2017-06-13 20:29:45,862 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.25 MB/134477608, currentsize=585.55 KB/599608 for region ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. in 365ms, sequenceid=2082447, compaction requested=true
2017-06-13 20:29:45,864 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789.
2017-06-13 20:29:45,864 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/.tmp, totalSize=2.4 M
2017-06-13 20:29:45,866 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61086, currentSize=4059769936, freeSize=213727920, maxSize=4273497856, heapSize=4059769936, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:29:46,129 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 (all) file(s) in info of ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. into 307db15e2f7946e78f3f69227d743ab5(size=1.0 M), total size for store is 1.0 M. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 20:29:46,129 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789., storeName=info, fileCount=3, fileSize=2.4 M, priority=7, time=21470548181524914; duration=0sec
2017-06-13 20:30:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.83 MB, max=3.98 GB, blockCount=61086, accesses=203973970, hits=104297021, hitRatio=51.13%, , cachingAccesses=200506983, cachingHits=104271497, cachingHitsRatio=52.00%, evictions=42051, evicted=94389124, evictedPerRun=2244.63427734375
2017-06-13 20:31:31,907 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.33 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:31:32,704 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36725247, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/c76e15ddb12646bca03d7fa13ca26e3e
2017-06-13 20:31:32,721 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/c76e15ddb12646bca03d7fa13ca26e3e, entries=297093, sequenceid=36725247, filesize=13.5 M
2017-06-13 20:31:32,722 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.33 MB/134567408, currentsize=783.78 KB/802592 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 815ms, sequenceid=36725247, compaction requested=true
2017-06-13 20:31:32,724 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:31:32,724 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 20:31:32,725 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61086, currentSize=4059769936, freeSize=213727920, maxSize=4273497856, heapSize=4059769936, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:31:34,771 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 1d2adcf06f154aef993f374e24947930(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 20:31:34,771 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21470655041533726; duration=2sec
2017-06-13 20:33:32,350 INFO  [sync.1] wal.FSHLog: Slow sync cost: 2949 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK]]
2017-06-13 20:33:32,408 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497385771002 with entries=496, filesize=121.97 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386012351
2017-06-13 20:34:48,271 INFO  [sync.4] wal.FSHLog: Slow sync cost: 1409 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK]]
2017-06-13 20:35:21,801 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:35:22,749 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36725789, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/ebf7c26c053e451698440e01de1aa236
2017-06-13 20:35:22,773 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/ebf7c26c053e451698440e01de1aa236, entries=297093, sequenceid=36725789, filesize=13.5 M
2017-06-13 20:35:22,774 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232744, currentsize=13.47 MB/14122032 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 973ms, sequenceid=36725789, compaction requested=true
2017-06-13 20:35:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.83 MB, max=3.98 GB, blockCount=61086, accesses=203976219, hits=104297057, hitRatio=51.13%, , cachingAccesses=200507019, cachingHits=104271533, cachingHitsRatio=52.00%, evictions=42081, evicted=94389124, evictedPerRun=2243.0341796875
2017-06-13 20:37:11,188 INFO  [sync.4] wal.FSHLog: Slow sync cost: 223 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK]]
2017-06-13 20:37:18,859 INFO  [sync.0] wal.FSHLog: Slow sync cost: 1902 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK]]
2017-06-13 20:39:08,025 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.30 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:39:09,073 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386012351 with entries=454, filesize=121.69 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386349038
2017-06-13 20:39:09,095 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36726328, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/558e69be616b4a76be3d752e436d189f
2017-06-13 20:39:09,108 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/558e69be616b4a76be3d752e436d189f, entries=297093, sequenceid=36726328, filesize=13.5 M
2017-06-13 20:39:09,110 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.30 MB/134531800, currentsize=10.76 MB/11280472 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1085ms, sequenceid=36726328, compaction requested=true
2017-06-13 20:39:09,112 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:39:09,113 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 20:39:09,114 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61086, currentSize=4059769936, freeSize=213727920, maxSize=4273497856, heapSize=4059769936, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:39:11,380 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 301827dcbb544e9d8555953a998372a7(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 20:39:11,380 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21471111430175247; duration=2sec
2017-06-13 20:40:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.83 MB, max=3.98 GB, blockCount=61086, accesses=203979968, hits=104297057, hitRatio=51.13%, , cachingAccesses=200507019, cachingHits=104271533, cachingHitsRatio=52.00%, evictions=42111, evicted=94389124, evictedPerRun=2241.436279296875
2017-06-13 20:41:07,721 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.05 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:41:08,507 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36726872, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/0214829a10be40cdb26459544d934c89
2017-06-13 20:41:08,521 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/0214829a10be40cdb26459544d934c89, entries=297093, sequenceid=36726872, filesize=13.5 M
2017-06-13 20:41:08,522 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.05 MB/134272664, currentsize=12.20 MB/12794464 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 801ms, sequenceid=36726872, compaction requested=true
2017-06-13 20:41:10,114 INFO  [sync.1] wal.FSHLog: Slow sync cost: 1196 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 20:41:35,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. after a delay of 19903
2017-06-13 20:41:45,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. after a delay of 4638
2017-06-13 20:41:55,492 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c., current region memstore size 37.45 KB, and 2/2 column families' memstores are being flushed.
2017-06-13 20:41:55,525 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11083, memsize=17.0 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp/3f09cf6b03884c71ac36ea4cd9dd4222
2017-06-13 20:41:55,554 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11083, memsize=20.5 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp/0d87b32f068945d98d1e7a66af9f688b
2017-06-13 20:41:55,566 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/ch/3f09cf6b03884c71ac36ea4cd9dd4222, entries=87, sequenceid=11083, filesize=5.7 K
2017-06-13 20:41:55,577 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/info/0d87b32f068945d98d1e7a66af9f688b, entries=102, sequenceid=11083, filesize=6.6 K
2017-06-13 20:41:55,578 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~37.45 KB/38344, currentsize=0 B/0 for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. in 86ms, sequenceid=11083, compaction requested=false
2017-06-13 20:42:57,237 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386349038 with entries=485, filesize=121.74 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386577207
2017-06-13 20:42:57,239 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497382774031 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497382774031
2017-06-13 20:42:57,251 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384015123 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497384015123
2017-06-13 20:42:57,266 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384370428 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497384370428
2017-06-13 20:44:41,688 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.41 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:44:42,482 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36727405, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/3f8364d164f64d51a5420c10751bbd56
2017-06-13 20:44:42,511 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/3f8364d164f64d51a5420c10751bbd56, entries=297093, sequenceid=36727405, filesize=13.5 M
2017-06-13 20:44:42,512 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.41 MB/134651736, currentsize=6.42 MB/6736120 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 824ms, sequenceid=36727405, compaction requested=true
2017-06-13 20:44:42,518 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:44:42,518 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 20:44:42,519 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61086, currentSize=4059769936, freeSize=213727920, maxSize=4273497856, heapSize=4059769936, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:44:44,381 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 5522aa80cdc74f1ca0cfc548ee0da273(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 20:44:44,381 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21471444835829673; duration=1sec
2017-06-13 20:45:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.83 MB, max=3.98 GB, blockCount=61086, accesses=203982181, hits=104297057, hitRatio=51.13%, , cachingAccesses=200507019, cachingHits=104271533, cachingHitsRatio=52.00%, evictions=42141, evicted=94389124, evictedPerRun=2239.840576171875
2017-06-13 20:46:49,128 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.38 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:46:51,111 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36727957, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/977be03fcc334c5eaea007da1ff1906c
2017-06-13 20:46:51,128 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/977be03fcc334c5eaea007da1ff1906c, entries=297093, sequenceid=36727957, filesize=13.5 M
2017-06-13 20:46:51,128 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.38 MB/134614016, currentsize=21.72 MB/22774576 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 2000ms, sequenceid=36727957, compaction requested=true
2017-06-13 20:48:43,693 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386577207 with entries=472, filesize=122.01 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386923574
2017-06-13 20:50:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.83 MB, max=3.98 GB, blockCount=61086, accesses=203982181, hits=104297057, hitRatio=51.13%, , cachingAccesses=200507019, cachingHits=104271533, cachingHitsRatio=52.00%, evictions=42171, evicted=94389124, evictedPerRun=2238.2470703125
2017-06-13 20:50:32,878 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.34 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:50:33,686 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36728480, memsize=128.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/b387a5ea0088423eaf00871da1d01354
2017-06-13 20:50:33,705 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/b387a5ea0088423eaf00871da1d01354, entries=297093, sequenceid=36728480, filesize=13.5 M
2017-06-13 20:50:33,706 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.34 MB/134574392, currentsize=10.06 MB/10543480 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 828ms, sequenceid=36728480, compaction requested=true
2017-06-13 20:50:33,711 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:50:33,711 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=40.4 M
2017-06-13 20:50:33,712 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61086, currentSize=4059769936, freeSize=213727920, maxSize=4273497856, heapSize=4059769936, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:50:35,621 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 73e1356f372b4d919e2e60a51caab4ba(size=13.5 M), total size for store is 188.5 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 20:50:35,621 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=40.4 M, priority=6, time=21471796028504566; duration=1sec
2017-06-13 20:51:05,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. after a delay of 8470
2017-06-13 20:51:14,060 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228., current region memstore size 44.39 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:51:14,363 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=266929, memsize=44.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/.tmp/ffc853b920ae4bd98eb2a0cc1f79ddee
2017-06-13 20:51:14,377 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/info/ffc853b920ae4bd98eb2a0cc1f79ddee, entries=223888, sequenceid=266929, filesize=2.8 M
2017-06-13 20:51:14,379 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~44.39 MB/46544888, currentsize=0 B/0 for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. in 319ms, sequenceid=266929, compaction requested=true
2017-06-13 20:51:14,381 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228.
2017-06-13 20:51:14,381 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/.tmp, totalSize=20.5 M
2017-06-13 20:51:14,393 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61086, currentSize=4059769936, freeSize=213727920, maxSize=4273497856, heapSize=4059769936, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:51:17,151 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. into e7e09d6a5edb45a89836148f33744ddc(size=17.8 M), total size for store is 303.0 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 20:51:17,151 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228., storeName=info, fileCount=3, fileSize=20.5 M, priority=6, time=21471836698346683; duration=2sec
2017-06-13 20:52:31,834 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386923574 with entries=470, filesize=121.86 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387151782
2017-06-13 20:52:31,835 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383239374 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497383239374
2017-06-13 20:52:31,853 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497383791680 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497383791680
2017-06-13 20:52:33,174 INFO  [sync.0] wal.FSHLog: Slow sync cost: 101 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-8ab76f9c-ee05-4ec0-897a-8718ab89635f,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-120b90e1-5d80-447d-b8db-b8ea53661e88,DISK]]
2017-06-13 20:52:33,636 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.02 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:52:34,630 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36729040, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/2ae9b8041b2246679c31d78126f7bdf4
2017-06-13 20:52:34,643 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/2ae9b8041b2246679c31d78126f7bdf4, entries=297093, sequenceid=36729040, filesize=13.5 M
2017-06-13 20:52:34,644 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.02 MB/134241048, currentsize=11.96 MB/12541520 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1009ms, sequenceid=36729040, compaction requested=true
2017-06-13 20:55:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.83 MB, max=3.98 GB, blockCount=61086, accesses=203986594, hits=104297057, hitRatio=51.13%, , cachingAccesses=200507019, cachingHits=104271533, cachingHitsRatio=52.00%, evictions=42201, evicted=94389124, evictedPerRun=2236.656005859375
2017-06-13 20:56:08,629 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.35 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:56:09,509 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36729553, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/c39508f62f20433da307d86da41e9170
2017-06-13 20:56:09,523 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/c39508f62f20433da307d86da41e9170, entries=299608, sequenceid=36729553, filesize=14.3 M
2017-06-13 20:56:09,524 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.35 MB/134588672, currentsize=12.70 MB/13315480 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 895ms, sequenceid=36729553, compaction requested=true
2017-06-13 20:56:09,526 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 20:56:09,526 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=41.2 M
2017-06-13 20:56:09,526 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61086, currentSize=4059769936, freeSize=213727920, maxSize=4273497856, heapSize=4059769936, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:56:11,498 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 96f5d988e0f645e3a1a270beb585a203(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 20:56:11,498 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=41.2 M, priority=6, time=21472131843385393; duration=1sec
2017-06-13 20:56:12,967 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387151782 with entries=472, filesize=121.69 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387372916
2017-06-13 20:58:09,072 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:58:10,097 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36730109, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/6c3aadc7ce5b46d0a02f269fb90162fe
2017-06-13 20:58:10,119 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/6c3aadc7ce5b46d0a02f269fb90162fe, entries=299608, sequenceid=36730109, filesize=14.3 M
2017-06-13 20:58:10,120 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=9.57 MB/10031064 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1048ms, sequenceid=36730109, compaction requested=true
2017-06-13 20:58:37,616 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 130.09 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:58:38,542 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1694528, memsize=132.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/02a37a2dc12c4e2fb34f35e2842f8459
2017-06-13 20:58:38,554 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/02a37a2dc12c4e2fb34f35e2842f8459, entries=776640, sequenceid=1694528, filesize=7.2 M
2017-06-13 20:58:38,555 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~132.61 MB/139048024, currentsize=31.40 MB/32924344 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 939ms, sequenceid=1694528, compaction requested=true
2017-06-13 20:58:38,556 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.
2017-06-13 20:58:38,556 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp, totalSize=7.3 M
2017-06-13 20:58:38,562 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61086, currentSize=4059769936, freeSize=213727920, maxSize=4273497856, heapSize=4059769936, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 20:58:39,540 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into 59c8811bb9204604aa828042ae4af483(size=7.2 M), total size for store is 3.6 G. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 20:58:39,540 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., storeName=info, fileCount=3, fileSize=7.3 M, priority=6, time=21472280874065186; duration=0sec
2017-06-13 20:58:41,675 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387372916 with entries=307, filesize=121.97 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387521633
2017-06-13 20:59:06,798 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 129.81 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 20:59:07,716 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1694638, memsize=129.8 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/e75b4841856944e7a5581a4d58664014
2017-06-13 20:59:07,728 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/e75b4841856944e7a5581a4d58664014, entries=760078, sequenceid=1694638, filesize=7.1 M
2017-06-13 20:59:07,729 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~129.81 MB/136115608, currentsize=18.68 MB/19583304 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 931ms, sequenceid=1694638, compaction requested=true
2017-06-13 21:00:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.83 MB, max=3.98 GB, blockCount=61086, accesses=203989447, hits=104297057, hitRatio=51.13%, , cachingAccesses=200507019, cachingHits=104271533, cachingHitsRatio=52.00%, evictions=42231, evicted=94389124, evictedPerRun=2235.067138671875
2017-06-13 21:02:11,105 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:02:11,972 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36730616, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/302e9b52f3924118b571a7338b143da6
2017-06-13 21:02:11,992 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/302e9b52f3924118b571a7338b143da6, entries=299608, sequenceid=36730616, filesize=14.3 M
2017-06-13 21:02:11,993 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=7.31 MB/7663800 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 888ms, sequenceid=36730616, compaction requested=true
2017-06-13 21:02:11,995 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 21:02:11,995 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 21:02:11,996 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=63348, currentSize=4211058792, freeSize=62439064, maxSize=4273497856, heapSize=4211058792, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:02:14,232 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 9d7cadafd3fe4d46b31f66d64e191db2(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 21:02:14,232 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21472494312688913; duration=2sec
2017-06-13 21:02:15,154 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387521633 with entries=401, filesize=121.94 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387735099
2017-06-13 21:02:18,153 INFO  [sync.4] wal.FSHLog: Slow sync cost: 875 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 21:03:35,396 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789., current region memstore size 128.40 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:03:35,726 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2083037, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/.tmp/9590615d4ab64979b8e0516486caec11
2017-06-13 21:03:35,746 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/info/9590615d4ab64979b8e0516486caec11, entries=37341, sequenceid=2083037, filesize=726.5 K
2017-06-13 21:03:35,747 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.40 MB/134636768, currentsize=187.58 KB/192080 for region ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. in 351ms, sequenceid=2083037, compaction requested=false
2017-06-13 21:04:16,912 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:04:17,784 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36731172, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/044eb46d4f9548fa8cecdaa73aa68224
2017-06-13 21:04:17,798 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/044eb46d4f9548fa8cecdaa73aa68224, entries=299608, sequenceid=36731172, filesize=14.3 M
2017-06-13 21:04:17,799 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=10.29 MB/10787080 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 887ms, sequenceid=36731172, compaction requested=true
2017-06-13 21:04:45,713 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500., current region memstore size 128.41 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:04:46,316 INFO  [sync.1] wal.FSHLog: Slow sync cost: 179 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 21:04:46,726 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387735099 with entries=333, filesize=123.04 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387886687
2017-06-13 21:04:46,727 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386349038 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497386349038
2017-06-13 21:04:46,740 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387151782 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497387151782
2017-06-13 21:04:47,098 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1467496, memsize=133.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/68378a5fea855a228a5514a2cc696500/.tmp/2c6ac716917c482b94d5d11e35f2977f
2017-06-13 21:04:47,114 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/68378a5fea855a228a5514a2cc696500/info/2c6ac716917c482b94d5d11e35f2977f, entries=781378, sequenceid=1467496, filesize=7.3 M
2017-06-13 21:04:47,114 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~133.44 MB/139916896, currentsize=68.11 MB/71422632 for region SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500. in 1401ms, sequenceid=1467496, compaction requested=true
2017-06-13 21:05:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.89 GB, freeSize=88.52 MB, max=3.98 GB, blockCount=62882, accesses=204422964, hits=104526916, hitRatio=51.13%, , cachingAccesses=200938290, cachingHits=104501392, cachingHitsRatio=52.01%, evictions=42314, evicted=94579371, evictedPerRun=2235.17919921875
2017-06-13 21:05:43,869 INFO  [RS_OPEN_META-aps-hadoop4:16020-0-MetaLogRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497384343708.meta with entries=0, filesize=91 B; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497387943800.meta
2017-06-13 21:05:43,869 INFO  [RS_OPEN_META-aps-hadoop4:16020-0-MetaLogRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497384343708.meta to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004..meta.1497384343708.meta
2017-06-13 21:05:53,503 WARN  [B.defaultRpcServer.handler=45,queue=0,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497387924944,"responsesize":415,"method":"Scan","processingtimems":28559,"client":"hadoop4:44288","queuetimems":0,"class":"HRegionServer"}
2017-06-13 21:05:53,504 WARN  [B.defaultRpcServer.handler=4,queue=4,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497387924944,"responsesize":415,"method":"Scan","processingtimems":28560,"client":"hadoop4:44302","queuetimems":0,"class":"HRegionServer"}
2017-06-13 21:07:17,647 INFO  [sync.1] wal.FSHLog: Slow sync cost: 1030 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK]]
2017-06-13 21:10:22,120 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:10:22,931 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36731679, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/f7e9325bc1444a749d9875a71d245805
2017-06-13 21:10:22,948 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/f7e9325bc1444a749d9875a71d245805, entries=299608, sequenceid=36731679, filesize=14.3 M
2017-06-13 21:10:22,950 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=9.37 MB/9824184 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 830ms, sequenceid=36731679, compaction requested=true
2017-06-13 21:10:22,952 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 21:10:22,952 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 21:10:22,953 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=63658, currentSize=4222485456, freeSize=51012400, maxSize=4273497856, heapSize=4222485456, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:10:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.91 GB, freeSize=69.87 MB, max=3.98 GB, blockCount=63319, accesses=205557038, hits=105163724, hitRatio=51.16%, , cachingAccesses=202070118, cachingHits=105138200, cachingHitsRatio=52.03%, evictions=42472, evicted=94986958, evictedPerRun=2236.460693359375
2017-06-13 21:10:25,012 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 17e332dacece4e7dbb5cee43b4cbff37(size=14.3 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 21:10:25,012 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21472985269824897; duration=2sec
2017-06-13 21:12:49,305 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387886687 with entries=442, filesize=122.42 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497388369200
2017-06-13 21:12:53,450 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:12:54,474 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36732235, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/d5cba8a479f347bc83c606e270da7cdc
2017-06-13 21:12:54,528 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/d5cba8a479f347bc83c606e270da7cdc, entries=299608, sequenceid=36732235, filesize=14.3 M
2017-06-13 21:12:54,531 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=10.29 MB/10787080 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1081ms, sequenceid=36732235, compaction requested=true
2017-06-13 21:14:12,428 INFO  [sync.0] wal.FSHLog: Slow sync cost: 2129 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:15:15,744 INFO  [sync.1] wal.FSHLog: Slow sync cost: 389 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:15:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.83 GB, freeSize=157.38 MB, max=3.98 GB, blockCount=61769, accesses=207386139, hits=106191376, hitRatio=51.20%, , cachingAccesses=203899219, cachingHits=106165852, cachingHitsRatio=52.07%, evictions=42732, evicted=95658096, evictedPerRun=2238.558837890625
2017-06-13 21:15:35,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. after a delay of 8225
2017-06-13 21:15:43,814 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf., current region memstore size 142.51 KB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:15:43,849 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=534439, memsize=142.5 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/.tmp/2c7695629cef43819c0ff5940098553e
2017-06-13 21:15:43,867 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/info/2c7695629cef43819c0ff5940098553e, entries=198, sequenceid=534439, filesize=8.9 K
2017-06-13 21:16:43,702 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082743103_9035751] hdfs.DFSClient: Slow ReadProcessor read fields took 59885ms (threshold=30000ms); ack: seqno: 1787 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 2512525 flag: 0 flag: 0 flag: 0, targets: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:16:43,702 WARN  [sync.1] hdfs.DFSClient: Slow waitForAckedSeqno took 59835ms (threshold=30000ms)
2017-06-13 21:16:43,702 INFO  [sync.1] wal.FSHLog: Slow sync cost: 59835 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:16:43,702 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~142.51 KB/145928, currentsize=0 B/0 for region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. in 59888ms, sequenceid=534439, compaction requested=false
2017-06-13 21:17:11,138 WARN  [sync.2] hdfs.DFSClient: Slow waitForAckedSeqno took 63413ms (threshold=30000ms)
2017-06-13 21:17:11,138 INFO  [sync.2] wal.FSHLog: Slow sync cost: 63412 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:17:11,138 WARN  [B.defaultRpcServer.handler=42,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1497388567724,"responsesize":8,"method":"Multi","processingtimems":63414,"client":"hadoop3:55124","queuetimems":0,"class":"HRegionServer"}
2017-06-13 21:17:11,138 INFO  [sync.4] wal.FSHLog: Slow sync cost: 15472 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:17:11,138 WARN  [sync.3] hdfs.DFSClient: Slow waitForAckedSeqno took 40853ms (threshold=30000ms)
2017-06-13 21:17:11,138 WARN  [B.defaultRpcServer.handler=47,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1497388615666,"responsesize":8,"method":"Multi","processingtimems":15472,"client":"hadoop3:55124","queuetimems":0,"class":"HRegionServer"}
2017-06-13 21:17:11,138 INFO  [sync.3] wal.FSHLog: Slow sync cost: 40852 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:17:11,139 WARN  [B.defaultRpcServer.handler=14,queue=4,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1497388590256,"responsesize":603,"method":"Multi","processingtimems":40882,"client":"hadoop3:55124","queuetimems":0,"class":"HRegionServer"}
2017-06-13 21:18:55,144 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:18:58,531 INFO  [sync.3] wal.FSHLog: Slow sync cost: 753 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:18:59,045 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36732742, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/fdbe3b8883c14b71a95e03d5d3cfb6c9
2017-06-13 21:18:59,058 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 1665 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:18:59,077 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/fdbe3b8883c14b71a95e03d5d3cfb6c9, entries=299608, sequenceid=36732742, filesize=14.3 M
2017-06-13 21:18:59,444 INFO  [sync.0] wal.FSHLog: Slow sync cost: 386 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:18:59,444 INFO  [sync.4] wal.FSHLog: Slow sync cost: 861 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:18:59,468 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497388369200 with entries=479, filesize=124.09 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497388737312
2017-06-13 21:18:59,471 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384601931 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497384601931
2017-06-13 21:18:59,474 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=29.86 MB/31315688 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 4330ms, sequenceid=36732742, compaction requested=true
2017-06-13 21:18:59,477 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 21:18:59,477 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 21:18:59,477 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=60953, currentSize=4068745624, freeSize=204752232, maxSize=4273497856, heapSize=4068745624, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:18:59,530 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497384952168 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497384952168
2017-06-13 21:18:59,559 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497385197787 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497385197787
2017-06-13 21:18:59,578 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497385437346 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497385437346
2017-06-13 21:18:59,594 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497385771002 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497385771002
2017-06-13 21:18:59,612 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386012351 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497386012351
2017-06-13 21:18:59,633 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387372916 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497387372916
2017-06-13 21:19:01,593 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 98108b7dbbb449b28d15ddb2c2ad9b73(size=14.3 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 21:19:01,593 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21473501794312730; duration=2sec
2017-06-13 21:19:02,826 INFO  [sync.4] wal.FSHLog: Slow sync cost: 244 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 21:19:14,604 INFO  [sync.3] wal.FSHLog: Slow sync cost: 450 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 21:20:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.91 GB, freeSize=67.95 MB, max=3.98 GB, blockCount=62967, accesses=208947185, hits=107000282, hitRatio=51.21%, , cachingAccesses=205458019, cachingHits=106974758, cachingHitsRatio=52.07%, evictions=43022, evicted=96406791, evictedPerRun=2240.871826171875
2017-06-13 21:21:07,613 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:21:08,652 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36733298, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/604108cc79da48bdb64d7e6c0d1ea730
2017-06-13 21:21:08,666 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/604108cc79da48bdb64d7e6c0d1ea730, entries=299608, sequenceid=36733298, filesize=14.3 M
2017-06-13 21:21:08,669 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=10.29 MB/10787080 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1056ms, sequenceid=36733298, compaction requested=true
2017-06-13 21:23:22,052 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497388737312 with entries=460, filesize=122.53 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389001942
2017-06-13 21:25:21,304 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:25:22,228 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36733805, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/024b7a4b0b3146f49901eff5aa1ea15f
2017-06-13 21:25:22,254 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/024b7a4b0b3146f49901eff5aa1ea15f, entries=299608, sequenceid=36733805, filesize=14.3 M
2017-06-13 21:25:22,255 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=7.73 MB/8101888 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 951ms, sequenceid=36733805, compaction requested=true
2017-06-13 21:25:22,259 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 21:25:22,259 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 21:25:22,259 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61816, currentSize=4129632200, freeSize=143865656, maxSize=4273497856, heapSize=4129632200, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:25:24,365 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 901a508467114057a36db1bdc63f923e(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 21:25:24,365 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21473884576443793; duration=2sec
2017-06-13 21:25:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.84 GB, freeSize=143.46 MB, max=3.98 GB, blockCount=61716, accesses=210433425, hits=107742049, hitRatio=51.20%, , cachingAccesses=206942013, cachingHits=107716525, cachingHitsRatio=52.05%, evictions=43310, evicted=97150268, evictedPerRun=2243.13720703125
2017-06-13 21:27:32,280 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:27:33,417 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36734361, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/b2fbe1813ac3456eb775fecc6169b44d
2017-06-13 21:27:33,436 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/b2fbe1813ac3456eb775fecc6169b44d, entries=299608, sequenceid=36734361, filesize=14.3 M
2017-06-13 21:27:33,442 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=9.57 MB/10031064 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1162ms, sequenceid=36734361, compaction requested=true
2017-06-13 21:29:50,305 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 102 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 21:29:50,334 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389001942 with entries=452, filesize=122.36 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389390188
2017-06-13 21:30:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.86 GB, freeSize=119.06 MB, max=3.98 GB, blockCount=62001, accesses=212239352, hits=108669883, hitRatio=51.20%, , cachingAccesses=208747940, cachingHits=108644359, cachingHitsRatio=52.05%, evictions=43650, evicted=98028078, evictedPerRun=2245.775146484375
2017-06-13 21:31:19,334 INFO  [sync.1] wal.FSHLog: Slow sync cost: 347 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK]]
2017-06-13 21:32:00,064 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:32:01,063 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36734868, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/b6cdb58caa4c4428afd1487e1a644f33
2017-06-13 21:32:01,081 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/b6cdb58caa4c4428afd1487e1a644f33, entries=299608, sequenceid=36734868, filesize=14.3 M
2017-06-13 21:32:01,087 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=11.45 MB/12006280 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1023ms, sequenceid=36734868, compaction requested=true
2017-06-13 21:32:01,089 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 21:32:01,089 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 21:32:01,090 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61568, currentSize=4121771880, freeSize=151725976, maxSize=4273497856, heapSize=4121771880, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:32:04,709 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 6f32c74e71914659a4f8f78087ef51ec(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 3sec to execute.
2017-06-13 21:32:04,709 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21474283407001867; duration=3sec
2017-06-13 21:32:06,381 INFO  [sync.2] wal.FSHLog: Slow sync cost: 1281 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK]]
2017-06-13 21:34:50,345 INFO  [sync.1] wal.FSHLog: Slow sync cost: 356 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK]]
2017-06-13 21:35:00,719 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389390188 with entries=481, filesize=122.41 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389700513
2017-06-13 21:35:01,252 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:35:02,215 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36735424, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/f6e88d49e4d24b50bf66e0b224304991
2017-06-13 21:35:02,240 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/f6e88d49e4d24b50bf66e0b224304991, entries=299608, sequenceid=36735424, filesize=14.3 M
2017-06-13 21:35:02,243 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=9.57 MB/10031064 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 991ms, sequenceid=36735424, compaction requested=true
2017-06-13 21:35:13,164 INFO  [sync.4] wal.FSHLog: Slow sync cost: 204 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:35:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.83 GB, freeSize=154.41 MB, max=3.98 GB, blockCount=61400, accesses=213978933, hits=109542827, hitRatio=51.19%, , cachingAccesses=210485275, cachingHits=109517303, cachingHitsRatio=52.03%, evictions=43985, evicted=98893067, evictedPerRun=2248.336181640625
2017-06-13 21:37:03,828 INFO  [sync.1] wal.FSHLog: Slow sync cost: 352 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:38:57,763 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:38:58,696 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36735931, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/372b11c5db464d0fa63c0edb03863b23
2017-06-13 21:38:58,727 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/372b11c5db464d0fa63c0edb03863b23, entries=299608, sequenceid=36735931, filesize=14.3 M
2017-06-13 21:38:58,730 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=9.78 MB/10257096 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 967ms, sequenceid=36735931, compaction requested=true
2017-06-13 21:38:58,736 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 21:38:58,736 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 21:38:58,737 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=60853, currentSize=4074898744, freeSize=198599112, maxSize=4273497856, heapSize=4074898744, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:39:01,403 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into f956e35f215148e187fe2cb8f35f5d34(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 21:39:01,403 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21474701054091925; duration=2sec
2017-06-13 21:39:10,900 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 5417 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK]]
2017-06-13 21:39:10,937 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389700513 with entries=450, filesize=122.64 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389945456
2017-06-13 21:40:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.90 GB, freeSize=78.95 MB, max=3.98 GB, blockCount=62599, accesses=216254133, hits=110701813, hitRatio=51.19%, , cachingAccesses=212758229, cachingHits=110676289, cachingHitsRatio=52.02%, evictions=44416, evicted=100005838, evictedPerRun=2251.572509765625
2017-06-13 21:41:05,629 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:41:06,777 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36736487, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/4f271ec84ab440d8aeaed854411ae929
2017-06-13 21:41:06,831 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/4f271ec84ab440d8aeaed854411ae929, entries=299608, sequenceid=36736487, filesize=14.3 M
2017-06-13 21:41:06,888 INFO  [sync.0] wal.FSHLog: Slow sync cost: 235 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-8ab76f9c-ee05-4ec0-897a-8718ab89635f,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK]]
2017-06-13 21:41:06,890 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=10.29 MB/10787080 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1261ms, sequenceid=36736487, compaction requested=true
2017-06-13 21:45:16,914 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:45:17,308 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389945456 with entries=455, filesize=121.82 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497390317239
2017-06-13 21:45:17,893 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36736994, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/96014f51921641a4b7b8fa2714677763
2017-06-13 21:45:17,909 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/96014f51921641a4b7b8fa2714677763, entries=299608, sequenceid=36736994, filesize=14.3 M
2017-06-13 21:45:17,911 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=11.45 MB/12006280 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 998ms, sequenceid=36736994, compaction requested=true
2017-06-13 21:45:17,913 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 21:45:17,913 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 21:45:17,914 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62359, currentSize=4174672704, freeSize=98825152, maxSize=4273497856, heapSize=4174672704, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:45:20,182 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into d1a46ff545b04a95b7b6165557d0436a(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 21:45:20,182 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21475080230563781; duration=2sec
2017-06-13 21:45:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.93 GB, freeSize=53.57 MB, max=3.98 GB, blockCount=63003, accesses=218369429, hits=111794587, hitRatio=51.20%, , cachingAccesses=214871279, cachingHits=111769063, cachingHitsRatio=52.02%, evictions=44811, evicted=101025709, evictedPerRun=2254.484619140625
2017-06-13 21:45:32,706 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789., current region memstore size 128.35 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:45:33,231 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2083632, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/.tmp/73ef1ef75aa54d07bf3037a07c28ac97
2017-06-13 21:45:33,251 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/info/73ef1ef75aa54d07bf3037a07c28ac97, entries=37341, sequenceid=2083632, filesize=727.7 K
2017-06-13 21:45:33,256 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.35 MB/134588776, currentsize=799.73 KB/818920 for region ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. in 550ms, sequenceid=2083632, compaction requested=true
2017-06-13 21:45:33,258 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789.
2017-06-13 21:45:33,258 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/.tmp, totalSize=2.4 M
2017-06-13 21:45:33,272 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62012, currentSize=4151677016, freeSize=121820840, maxSize=4273497856, heapSize=4151677016, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:45:33,593 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 (all) file(s) in info of ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. into 8618173ebbf5447f9c84d170461ff747(size=1.0 M), total size for store is 1.0 M. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 21:45:33,594 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789., storeName=info, fileCount=3, fileSize=2.4 M, priority=7, time=21475095575890929; duration=0sec
2017-06-13 21:47:15,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. after a delay of 21052
2017-06-13 21:47:25,214 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:47:25,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. after a delay of 14358
2017-06-13 21:47:26,414 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36737550, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/f3943db022ac4ae691858a40ae4a1e74
2017-06-13 21:47:26,429 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/f3943db022ac4ae691858a40ae4a1e74, entries=299608, sequenceid=36737550, filesize=14.3 M
2017-06-13 21:47:26,432 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=11.97 MB/12549416 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1218ms, sequenceid=36737550, compaction requested=true
2017-06-13 21:47:35,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. after a delay of 14199
2017-06-13 21:47:36,641 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c., current region memstore size 20.01 KB, and 2/2 column families' memstores are being flushed.
2017-06-13 21:47:36,751 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11104, memsize=9.2 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp/a1383ca17724424da2def9cf0c6e2767
2017-06-13 21:47:36,831 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11104, memsize=10.8 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp/30fac33ca1b5451d8c41dc9952b1ee18
2017-06-13 21:47:36,854 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/ch/a1383ca17724424da2def9cf0c6e2767, entries=47, sequenceid=11104, filesize=5.4 K
2017-06-13 21:47:36,886 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/info/30fac33ca1b5451d8c41dc9952b1ee18, entries=54, sequenceid=11104, filesize=5.8 K
2017-06-13 21:47:36,889 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~20.01 KB/20488, currentsize=0 B/0 for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. in 248ms, sequenceid=11104, compaction requested=true
2017-06-13 21:47:36,893 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on ch in region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c.
2017-06-13 21:47:36,894 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in ch of SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp, totalSize=70.9 K
2017-06-13 21:47:36,896 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62129, currentSize=4159395288, freeSize=114102568, maxSize=4273497856, heapSize=4159395288, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:47:37,122 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 (all) file(s) in ch of SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. into b9e204cd525142c18bde71b978601e3a(size=60.3 K), total size for store is 60.3 K. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 21:47:37,123 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c., storeName=ch, fileCount=3, fileSize=70.9 K, priority=7, time=21475219211253879; duration=0sec
2017-06-13 21:47:37,125 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c.
2017-06-13 21:47:37,125 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp, totalSize=108.2 K
2017-06-13 21:47:37,128 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=63162, currentSize=4227803528, freeSize=45694328, maxSize=4273497856, heapSize=4227803528, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:47:37,302 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 (all) file(s) in info of SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. into a1f879e3dd7a428f96331a55a04968da(size=96.9 K), total size for store is 96.9 K. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 21:47:37,302 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c., storeName=info, fileCount=3, fileSize=108.2 K, priority=7, time=21475219442399593; duration=0sec
2017-06-13 21:49:22,700 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497390317239 with entries=489, filesize=122.00 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497390562635
2017-06-13 21:49:22,704 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386577207 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497386577207
2017-06-13 21:49:22,727 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497386923574 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497386923574
2017-06-13 21:50:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.84 GB, freeSize=147.76 MB, max=3.98 GB, blockCount=61512, accesses=220589070, hits=112933700, hitRatio=51.20%, , cachingAccesses=217090744, cachingHits=112908176, cachingHitsRatio=52.01%, evictions=45230, evicted=102107555, evictedPerRun=2257.518310546875
2017-06-13 21:51:23,013 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:51:23,907 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36738057, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/7c9d42d35f2a4b44b9845f5ba10bc8b9
2017-06-13 21:51:23,925 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/7c9d42d35f2a4b44b9845f5ba10bc8b9, entries=299608, sequenceid=36738057, filesize=14.3 M
2017-06-13 21:51:23,933 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=8.55 MB/8966528 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 919ms, sequenceid=36738057, compaction requested=true
2017-06-13 21:51:23,935 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 21:51:23,935 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 21:51:23,937 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62497, currentSize=4183752848, freeSize=89745008, maxSize=4273497856, heapSize=4183752848, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:51:26,114 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 36d599356be34b32b8dd534249c9dbfd(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 21:51:26,114 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21475446253186125; duration=2sec
2017-06-13 21:51:30,732 INFO  [sync.4] wal.FSHLog: Slow sync cost: 148 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-3109a88b-277e-4d45-8cfb-8a5f3bf57adb,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK]]
2017-06-13 21:53:31,271 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:53:33,741 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36738613, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/b58ef7f43ec14d46b1227619f71aef07
2017-06-13 21:53:33,762 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/b58ef7f43ec14d46b1227619f71aef07, entries=299608, sequenceid=36738613, filesize=14.3 M
2017-06-13 21:53:33,770 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=20.75 MB/21760808 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 2499ms, sequenceid=36738613, compaction requested=true
2017-06-13 21:53:44,534 INFO  [sync.0] wal.FSHLog: Slow sync cost: 1370 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-3109a88b-277e-4d45-8cfb-8a5f3bf57adb,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK]]
2017-06-13 21:53:45,822 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497390562635 with entries=448, filesize=121.65 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497390825781
2017-06-13 21:55:24,520 INFO  [sync.1] wal.FSHLog: Slow sync cost: 1585 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-8ab76f9c-ee05-4ec0-897a-8718ab89635f,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 21:55:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.88 GB, freeSize=103.34 MB, max=3.98 GB, blockCount=62215, accesses=222501881, hits=113914024, hitRatio=51.20%, , cachingAccesses=219001309, cachingHits=113888500, cachingHitsRatio=52.00%, evictions=45590, evicted=103037093, evictedPerRun=2260.0810546875
2017-06-13 21:56:36,722 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 129.07 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:56:38,365 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1694785, memsize=129.1 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/b4bb248fd81b4452a57f987b20c987fb
2017-06-13 21:56:38,400 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/b4bb248fd81b4452a57f987b20c987fb, entries=755002, sequenceid=1694785, filesize=7.1 M
2017-06-13 21:56:38,403 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~129.07 MB/135343264, currentsize=22.64 MB/23742144 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 1681ms, sequenceid=1694785, compaction requested=true
2017-06-13 21:56:38,409 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.
2017-06-13 21:56:38,410 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp, totalSize=21.4 M
2017-06-13 21:56:38,426 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61544, currentSize=4102265776, freeSize=171232080, maxSize=4273497856, heapSize=4102265776, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:56:42,136 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into 2afd9343a7bd4b2cb05ced96f038748b(size=21.3 M), total size for store is 3.6 G. This selection was in queue for 0sec, and took 3sec to execute.
2017-06-13 21:56:42,136 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., storeName=info, fileCount=3, fileSize=21.4 M, priority=6, time=21475760727139977; duration=3sec
2017-06-13 21:56:42,476 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497390825781 with entries=296, filesize=122.13 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391002436
2017-06-13 21:56:46,167 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 128.51 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:56:47,113 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1694993, memsize=128.5 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/dba0f069de92496ab9fe286c0409d258
2017-06-13 21:56:47,156 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/dba0f069de92496ab9fe286c0409d258, entries=751254, sequenceid=1694993, filesize=7.1 M
2017-06-13 21:56:47,173 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.51 MB/134754400, currentsize=12.51 MB/13114288 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 1006ms, sequenceid=1694993, compaction requested=true
2017-06-13 21:56:54,546 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 128.63 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:56:55,480 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1695200, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/9b9e288e4aa34b7985b73ab9b4d49daf
2017-06-13 21:56:55,517 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/9b9e288e4aa34b7985b73ab9b4d49daf, entries=752036, sequenceid=1695200, filesize=7.1 M
2017-06-13 21:56:55,519 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.63 MB/134874560, currentsize=15.22 MB/15957088 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 973ms, sequenceid=1695200, compaction requested=true
2017-06-13 21:56:55,527 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.
2017-06-13 21:56:55,527 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp, totalSize=35.5 M
2017-06-13 21:56:55,528 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62034, currentSize=4135294176, freeSize=138203680, maxSize=4273497856, heapSize=4135294176, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:56:56,978 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391002436 with entries=175, filesize=121.96 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391016913
2017-06-13 21:57:00,210 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into e3c85366fc06455e960f4299cf84ddca(size=35.5 M), total size for store is 3.6 G. This selection was in queue for 0sec, and took 4sec to execute.
2017-06-13 21:57:00,210 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., storeName=info, fileCount=3, fileSize=35.5 M, priority=6, time=21475777844642711; duration=4sec
2017-06-13 21:57:09,020 INFO  [sync.3] wal.FSHLog: Slow sync cost: 1960 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 21:57:10,309 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 129.09 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:57:11,392 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1695410, memsize=129.1 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/3237ddccf6c44f7888810bf930a6d992
2017-06-13 21:57:11,425 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/3237ddccf6c44f7888810bf930a6d992, entries=754726, sequenceid=1695410, filesize=7.1 M
2017-06-13 21:57:11,426 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~129.09 MB/135357320, currentsize=16.31 MB/17099528 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 1117ms, sequenceid=1695410, compaction requested=true
2017-06-13 21:57:13,675 INFO  [sync.0] wal.FSHLog: Slow sync cost: 126 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 21:57:17,317 INFO  [sync.3] wal.FSHLog: Slow sync cost: 296 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 21:57:18,904 INFO  [sync.2] wal.FSHLog: Slow sync cost: 347 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 21:57:19,969 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 128.39 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:57:20,896 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 1509 ms, current pipeline: []
2017-06-13 21:57:21,181 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1695617, memsize=128.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/442b532fc9da4352b8b5a723e2d589fe
2017-06-13 21:57:21,206 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/442b532fc9da4352b8b5a723e2d589fe, entries=750884, sequenceid=1695617, filesize=7.1 M
2017-06-13 21:57:21,685 INFO  [sync.3] wal.FSHLog: Slow sync cost: 786 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-120b90e1-5d80-447d-b8db-b8ea53661e88,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK]]
2017-06-13 21:57:21,720 INFO  [sync.2] wal.FSHLog: Slow sync cost: 1591 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK]]
2017-06-13 21:57:21,723 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391016913 with entries=189, filesize=128.62 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391039371
2017-06-13 21:57:21,726 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391002436 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391002436
2017-06-13 21:57:21,727 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.39 MB/134623088, currentsize=2.52 MB/2646880 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 1758ms, sequenceid=1695617, compaction requested=true
2017-06-13 21:57:21,737 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.
2017-06-13 21:57:21,737 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp, totalSize=49.7 M
2017-06-13 21:57:21,738 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62179, currentSize=4145938376, freeSize=127559480, maxSize=4273497856, heapSize=4145938376, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:57:29,738 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 128.49 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:57:31,120 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into 51d91d96e53b4de69cad7b42aff4b709(size=49.7 M), total size for store is 3.7 G. This selection was in queue for 0sec, and took 9sec to execute.
2017-06-13 21:57:31,120 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., storeName=info, fileCount=3, fileSize=49.7 M, priority=6, time=21475804054658263; duration=9sec
2017-06-13 21:57:31,560 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1695826, memsize=129.8 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/08aadf9d86bb47338dcb853ee1f365df
2017-06-13 21:57:31,595 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/08aadf9d86bb47338dcb853ee1f365df, entries=758886, sequenceid=1695826, filesize=7.2 M
2017-06-13 21:57:31,624 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~129.80 MB/136105656, currentsize=42.80 MB/44883304 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 1886ms, sequenceid=1695826, compaction requested=true
2017-06-13 21:57:33,825 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391039371 with entries=177, filesize=122.32 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391053741
2017-06-13 21:57:33,936 INFO  [sync.0] wal.FSHLog: Slow sync cost: 100 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-3109a88b-277e-4d45-8cfb-8a5f3bf57adb,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 21:57:33,980 INFO  [sync.1] wal.FSHLog: Slow sync cost: 124 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-3109a88b-277e-4d45-8cfb-8a5f3bf57adb,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 21:57:35,786 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 128.61 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:57:36,768 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1696036, memsize=129.9 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/79075acf016e4b60af4940163052d220
2017-06-13 21:57:36,784 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/79075acf016e4b60af4940163052d220, entries=759156, sequenceid=1696036, filesize=7.2 M
2017-06-13 21:57:36,821 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~129.86 MB/136168400, currentsize=14.38 MB/15073680 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 1035ms, sequenceid=1696036, compaction requested=true
2017-06-13 21:57:36,825 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.
2017-06-13 21:57:36,825 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp, totalSize=64.1 M
2017-06-13 21:57:36,857 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=63259, currentSize=4217566248, freeSize=55931608, maxSize=4273497856, heapSize=4217566248, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:57:40,163 INFO  [sync.0] wal.FSHLog: Slow sync cost: 378 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-3109a88b-277e-4d45-8cfb-8a5f3bf57adb,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 21:57:43,462 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 129.23 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:57:44,984 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1696245, memsize=129.2 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/05e2cd2d2933477f8f4e192c1e7fc1b1
2017-06-13 21:57:44,998 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/05e2cd2d2933477f8f4e192c1e7fc1b1, entries=755584, sequenceid=1696245, filesize=7.2 M
2017-06-13 21:57:45,031 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~129.23 MB/135505144, currentsize=36.53 MB/38305656 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 1569ms, sequenceid=1696245, compaction requested=false
2017-06-13 21:57:45,216 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into 60e1cd8140d546439c3d5d44eb7c5504(size=64.0 M), total size for store is 3.7 G. This selection was in queue for 0sec, and took 8sec to execute.
2017-06-13 21:57:45,216 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., storeName=info, fileCount=3, fileSize=64.1 M, priority=6, time=21475819143060064; duration=8sec
2017-06-13 21:57:45,392 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391053741 with entries=178, filesize=122.24 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391065352
2017-06-13 21:57:47,356 INFO  [sync.3] wal.FSHLog: Slow sync cost: 204 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-6e0a3279-95d2-4bff-ac72-ed8e01ad47ab,DISK]]
2017-06-13 21:57:49,515 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 128.46 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:57:50,646 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1696455, memsize=129.7 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/ad6d86cff20e4fc6989f527772baac57
2017-06-13 21:57:50,662 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/ad6d86cff20e4fc6989f527772baac57, entries=758390, sequenceid=1696455, filesize=7.2 M
2017-06-13 21:57:50,663 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~129.72 MB/136022680, currentsize=22.62 MB/23723624 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 1148ms, sequenceid=1696455, compaction requested=true
2017-06-13 21:57:50,665 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.
2017-06-13 21:57:50,665 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp, totalSize=78.3 M
2017-06-13 21:57:50,666 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61995, currentSize=4133833968, freeSize=139663888, maxSize=4273497856, heapSize=4133833968, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:57:55,626 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 128.60 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:57:57,759 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 2155 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 21:57:57,814 INFO  [sync.0] wal.FSHLog: Slow sync cost: 1787 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 21:57:57,818 INFO  [sync.4] wal.FSHLog: Slow sync cost: 1819 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 21:57:57,839 INFO  [sync.1] wal.FSHLog: Slow sync cost: 1788 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 21:57:57,866 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391065352 with entries=194, filesize=129.47 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391075593
2017-06-13 21:57:58,040 INFO  [sync.0] wal.FSHLog: Slow sync cost: 106 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 21:57:58,099 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1696664, memsize=129.9 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/5f28e35654b241d68c8f5baee0ba4f9b
2017-06-13 21:57:58,119 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/5f28e35654b241d68c8f5baee0ba4f9b, entries=759134, sequenceid=1696664, filesize=7.2 M
2017-06-13 21:57:58,125 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~129.85 MB/136159640, currentsize=12.68 MB/13296056 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 2499ms, sequenceid=1696664, compaction requested=false
2017-06-13 21:58:00,516 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:58:01,384 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36739120, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/7adaa74070d94fffb51f9ca30a4c1b60
2017-06-13 21:58:01,401 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/7adaa74070d94fffb51f9ca30a4c1b60, entries=299608, sequenceid=36739120, filesize=14.3 M
2017-06-13 21:58:01,411 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=5.24 MB/5489856 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 895ms, sequenceid=36739120, compaction requested=true
2017-06-13 21:58:02,021 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into 9ff8cb55d21e4c6bb44ae3613cb0c153(size=78.2 M), total size for store is 3.7 G. This selection was in queue for 0sec, and took 11sec to execute.
2017-06-13 21:58:02,021 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., storeName=info, fileCount=3, fileSize=78.3 M, priority=6, time=21475832982969095; duration=11sec
2017-06-13 21:58:02,029 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 21:58:02,029 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 21:58:02,030 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61464, currentSize=4098707032, freeSize=174790824, maxSize=4273497856, heapSize=4098707032, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:58:04,063 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 3a0835c42b3941cdb90aeecabfb99be6(size=14.3 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 21:58:04,063 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21475844347151279; duration=2sec
2017-06-13 21:58:05,155 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 128.81 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:58:06,126 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1696874, memsize=128.8 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/72facdbf3ba14f5d9f6bfb2652852d48
2017-06-13 21:58:06,140 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/72facdbf3ba14f5d9f6bfb2652852d48, entries=752978, sequenceid=1696874, filesize=7.1 M
2017-06-13 21:58:06,152 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.81 MB/135069952, currentsize=12.74 MB/13359312 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 997ms, sequenceid=1696874, compaction requested=true
2017-06-13 21:58:06,155 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.
2017-06-13 21:58:06,155 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp, totalSize=92.5 M
2017-06-13 21:58:06,156 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62838, currentSize=4189832056, freeSize=83665800, maxSize=4273497856, heapSize=4189832056, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 21:58:09,131 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 1572 ms, current pipeline: []
2017-06-13 21:58:10,221 INFO  [sync.0] wal.FSHLog: Slow sync cost: 1089 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK]]
2017-06-13 21:58:10,221 INFO  [sync.4] wal.FSHLog: Slow sync cost: 2167 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK]]
2017-06-13 21:58:10,251 INFO  [sync.3] wal.FSHLog: Slow sync cost: 2270 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 21:58:10,252 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391075593 with entries=287, filesize=128.85 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391087547
2017-06-13 21:58:25,336 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into 61c19da681c748f681478bb0bbf85804(size=92.5 M), total size for store is 3.7 G. This selection was in queue for 0sec, and took 19sec to execute.
2017-06-13 21:58:25,336 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., storeName=info, fileCount=3, fileSize=92.5 M, priority=6, time=21475848472451758; duration=19sec
2017-06-13 21:59:35,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SMSCampaignStatus,zAcwE41OAPE2K6,1489986614646.18690781354093c773e1a351396d0479. after a delay of 5979
2017-06-13 21:59:41,568 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwE41OAPE2K6,1489986614646.18690781354093c773e1a351396d0479., current region memstore size 415.07 KB, and 1/1 column families' memstores are being flushed.
2017-06-13 21:59:41,609 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=861015, memsize=415.1 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/18690781354093c773e1a351396d0479/.tmp/6ba92afbd77f4f22a55d19743d271fc3
2017-06-13 21:59:41,624 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/18690781354093c773e1a351396d0479/info/6ba92afbd77f4f22a55d19743d271fc3, entries=2318, sequenceid=861015, filesize=35.7 K
2017-06-13 21:59:41,628 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~415.07 KB/425032, currentsize=0 B/0 for region SMSCampaignStatus,zAcwE41OAPE2K6,1489986614646.18690781354093c773e1a351396d0479. in 60ms, sequenceid=861015, compaction requested=false
2017-06-13 22:00:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.92 GB, freeSize=64.30 MB, max=3.98 GB, blockCount=63257, accesses=223571875, hits=114457579, hitRatio=51.19%, , cachingAccesses=220040539, cachingHits=114432055, cachingHitsRatio=52.00%, evictions=45777, evicted=103519323, evictedPerRun=2261.3828125
2017-06-13 22:00:35,628 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:00:36,890 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36739676, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/0bdaea9db77140dea18e4990108d0ae5
2017-06-13 22:00:36,906 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/0bdaea9db77140dea18e4990108d0ae5, entries=299608, sequenceid=36739676, filesize=14.3 M
2017-06-13 22:00:36,916 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=13.20 MB/13836808 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1288ms, sequenceid=36739676, compaction requested=true
2017-06-13 22:02:31,099 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 1315 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-6e0a3279-95d2-4bff-ac72-ed8e01ad47ab,DISK]]
2017-06-13 22:02:31,105 INFO  [sync.4] wal.FSHLog: Slow sync cost: 249 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-6e0a3279-95d2-4bff-ac72-ed8e01ad47ab,DISK]]
2017-06-13 22:02:31,147 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391087547 with entries=394, filesize=128.07 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391349771
2017-06-13 22:02:31,149 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387521633 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497387521633
2017-06-13 22:03:15,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SMSCampaignStatus,cccccccc,1493708419209.7d0daf3c7809bb67f79037b656436756. after a delay of 7649
2017-06-13 22:03:23,238 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,cccccccc,1493708419209.7d0daf3c7809bb67f79037b656436756., current region memstore size 4.37 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:03:24,690 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1046937, memsize=4.4 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/7d0daf3c7809bb67f79037b656436756/.tmp/c70da8f9062142f9be15f9872e773040
2017-06-13 22:03:24,735 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/7d0daf3c7809bb67f79037b656436756/info/c70da8f9062142f9be15f9872e773040, entries=25404, sequenceid=1046937, filesize=330.6 K
2017-06-13 22:03:24,739 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~4.37 MB/4581280, currentsize=0 B/0 for region SMSCampaignStatus,cccccccc,1493708419209.7d0daf3c7809bb67f79037b656436756. in 1501ms, sequenceid=1046937, compaction requested=true
2017-06-13 22:03:24,747 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,cccccccc,1493708419209.7d0daf3c7809bb67f79037b656436756.
2017-06-13 22:03:24,747 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,cccccccc,1493708419209.7d0daf3c7809bb67f79037b656436756. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/7d0daf3c7809bb67f79037b656436756/.tmp, totalSize=988.7 K
2017-06-13 22:03:24,791 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61905, currentSize=4123071824, freeSize=150426032, maxSize=4273497856, heapSize=4123071824, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:03:25,026 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,cccccccc,1493708419209.7d0daf3c7809bb67f79037b656436756. into 84fb06eeeb064293b5fe6bbd53cb2f88(size=326.9 K), total size for store is 638.8 M. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 22:03:25,026 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,cccccccc,1493708419209.7d0daf3c7809bb67f79037b656436756., storeName=info, fileCount=3, fileSize=988.7 K, priority=6, time=21476167064621769; duration=0sec
2017-06-13 22:04:26,715 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:04:27,586 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36740183, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/a99e9e9053ad41d4976d6a4df95efe79
2017-06-13 22:04:27,606 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/a99e9e9053ad41d4976d6a4df95efe79, entries=299608, sequenceid=36740183, filesize=14.3 M
2017-06-13 22:04:27,607 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=10.20 MB/10690376 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 892ms, sequenceid=36740183, compaction requested=true
2017-06-13 22:04:27,610 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 22:04:27,610 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 22:04:27,611 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61066, currentSize=4069746728, freeSize=203751128, maxSize=4273497856, heapSize=4069746728, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:04:29,805 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into c5d0aaa171de48f59a7646ebf6a2ab49(size=14.3 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 22:04:29,805 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21476229927865061; duration=2sec
2017-06-13 22:04:55,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500. after a delay of 15047
2017-06-13 22:05:05,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500. after a delay of 9475
2017-06-13 22:05:10,637 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500., current region memstore size 78.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:05:11,171 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1467567, memsize=78.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/68378a5fea855a228a5514a2cc696500/.tmp/9b248da09d4844ddbfee919ae7e89173
2017-06-13 22:05:11,186 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/68378a5fea855a228a5514a2cc696500/info/9b248da09d4844ddbfee919ae7e89173, entries=460096, sequenceid=1467567, filesize=4.3 M
2017-06-13 22:05:11,188 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~78.58 MB/82398888, currentsize=0 B/0 for region SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500. in 551ms, sequenceid=1467567, compaction requested=true
2017-06-13 22:05:11,190 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500.
2017-06-13 22:05:11,190 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/68378a5fea855a228a5514a2cc696500/.tmp, totalSize=12.5 M
2017-06-13 22:05:11,236 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62589, currentSize=4166175008, freeSize=107322848, maxSize=4273497856, heapSize=4166175008, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:05:12,908 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500. into 191d9340006c4d2c915d5c9dc0b49852(size=12.5 M), total size for store is 2.2 G. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 22:05:12,908 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500., storeName=info, fileCount=3, fileSize=12.5 M, priority=6, time=21476273508164729; duration=1sec
2017-06-13 22:05:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=200.12 MB, max=3.98 GB, blockCount=61043, accesses=224676067, hits=115024931, hitRatio=51.20%, , cachingAccesses=221140555, cachingHits=114999407, cachingHitsRatio=52.00%, evictions=45982, evicted=104048548, evictedPerRun=2262.810302734375
2017-06-13 22:05:44,255 INFO  [RS_OPEN_META-aps-hadoop4:16020-0-MetaLogRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497387943800.meta with entries=0, filesize=91 B; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497391544210.meta
2017-06-13 22:05:44,256 INFO  [RS_OPEN_META-aps-hadoop4:16020-0-MetaLogRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497387943800.meta to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004..meta.1497387943800.meta
2017-06-13 22:06:40,785 INFO  [sync.0] wal.FSHLog: Slow sync cost: 201 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-8ab76f9c-ee05-4ec0-897a-8718ab89635f,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 22:06:42,265 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:06:43,587 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36740739, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/da6956a95097471b9b743d2796a33c79
2017-06-13 22:06:43,606 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/da6956a95097471b9b743d2796a33c79, entries=299608, sequenceid=36740739, filesize=14.3 M
2017-06-13 22:06:43,607 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=13.95 MB/14626536 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1342ms, sequenceid=36740739, compaction requested=true
2017-06-13 22:06:44,265 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391349771 with entries=465, filesize=121.72 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391604208
2017-06-13 22:06:44,266 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387735099 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497387735099
2017-06-13 22:06:44,275 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497387886687 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497387886687
2017-06-13 22:10:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.94 GB, freeSize=43.21 MB, max=3.98 GB, blockCount=63652, accesses=225570235, hits=115505434, hitRatio=51.21%, , cachingAccesses=222026024, cachingHits=115479909, cachingHitsRatio=52.01%, evictions=46122, evicted=104409739, evictedPerRun=2263.77294921875
2017-06-13 22:10:42,922 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:10:43,777 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36741246, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/35ae359b946a4b70aa29d584c07c07b6
2017-06-13 22:10:43,795 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/35ae359b946a4b70aa29d584c07c07b6, entries=299608, sequenceid=36741246, filesize=14.3 M
2017-06-13 22:10:43,796 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=11.03 MB/11570960 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 874ms, sequenceid=36741246, compaction requested=true
2017-06-13 22:10:43,797 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 22:10:43,797 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 22:10:43,798 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61265, currentSize=4069830208, freeSize=203667648, maxSize=4273497856, heapSize=4069830208, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:10:45,878 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 4ac77f9ad36248bca4f2edce9107f2c9(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 22:10:45,878 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21476606115281218; duration=2sec
2017-06-13 22:12:43,143 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391604208 with entries=449, filesize=121.77 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391963091
2017-06-13 22:12:46,457 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:12:47,395 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36741802, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/773d4f8c21cd46a2be49d0f745ae3492
2017-06-13 22:12:47,408 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/773d4f8c21cd46a2be49d0f745ae3492, entries=299608, sequenceid=36741802, filesize=14.3 M
2017-06-13 22:12:47,409 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=10.29 MB/10787080 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 952ms, sequenceid=36741802, compaction requested=true
2017-06-13 22:12:59,000 INFO  [sync.4] wal.FSHLog: Slow sync cost: 549 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 22:14:40,004 INFO  [sync.2] wal.FSHLog: Slow sync cost: 6909 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 22:14:46,960 INFO  [sync.3] wal.FSHLog: Slow sync cost: 6834 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 22:14:49,219 INFO  [sync.0] wal.FSHLog: Slow sync cost: 741 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 22:14:50,444 INFO  [sync.2] wal.FSHLog: Slow sync cost: 116 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 22:15:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.82 GB, freeSize=168.84 MB, max=3.98 GB, blockCount=61661, accesses=225856644, hits=115662163, hitRatio=51.21%, , cachingAccesses=222310187, cachingHits=115636638, cachingHitsRatio=52.02%, evictions=46175, evicted=104539165, evictedPerRun=2263.9775390625
2017-06-13 22:16:15,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. after a delay of 14926
2017-06-13 22:16:25,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. after a delay of 18606
2017-06-13 22:16:30,515 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf., current region memstore size 148.29 KB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:16:30,538 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=534528, memsize=148.3 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/.tmp/4ef91748268b4a5587b3605b053d4d7c
2017-06-13 22:16:30,550 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/info/4ef91748268b4a5587b3605b053d4d7c, entries=264, sequenceid=534528, filesize=10.1 K
2017-06-13 22:16:30,551 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~148.29 KB/151848, currentsize=0 B/0 for region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. in 36ms, sequenceid=534528, compaction requested=true
2017-06-13 22:16:30,553 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf.
2017-06-13 22:16:30,553 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/.tmp, totalSize=522.2 K
2017-06-13 22:16:30,569 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62177, currentSize=4130312920, freeSize=143184936, maxSize=4273497856, heapSize=4130312920, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:16:31,002 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 (all) file(s) in info of CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. into 7bc058d0f63d475d99f77013a9605d77(size=503.3 K), total size for store is 503.3 K. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 22:16:31,002 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf., storeName=info, fileCount=3, fileSize=522.2 K, priority=7, time=21476952870708995; duration=0sec
2017-06-13 22:16:39,949 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:16:40,808 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36742309, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/3439e62a630b471d9832432c470ed363
2017-06-13 22:16:40,828 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/3439e62a630b471d9832432c470ed363, entries=299608, sequenceid=36742309, filesize=14.3 M
2017-06-13 22:16:40,830 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=11.03 MB/11570960 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 882ms, sequenceid=36742309, compaction requested=true
2017-06-13 22:16:40,832 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 22:16:40,832 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 22:16:40,833 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=63073, currentSize=4189781664, freeSize=83716192, maxSize=4273497856, heapSize=4189781664, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:16:41,867 INFO  [sync.3] wal.FSHLog: Slow sync cost: 676 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-485b95df-5888-4104-8455-5448e0f7846b,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 22:16:42,839 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 286681c8df7d4ce3bf1e063b58498df5(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 22:16:42,839 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21476963149618457; duration=2sec
2017-06-13 22:16:43,326 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391963091 with entries=478, filesize=121.98 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497392203267
2017-06-13 22:16:43,327 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497388369200 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497388369200
2017-06-13 22:16:43,355 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497388737312 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497388737312
2017-06-13 22:16:43,379 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389001942 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497389001942
2017-06-13 22:16:43,402 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389390188 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497389390188
2017-06-13 22:16:43,423 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389700513 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497389700513
2017-06-13 22:16:43,443 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497389945456 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497389945456
2017-06-13 22:16:43,470 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391053741 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391053741
2017-06-13 22:16:57,091 INFO  [sync.2] wal.FSHLog: Slow sync cost: 640 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK]]
2017-06-13 22:18:34,325 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:18:35,242 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36742865, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/0fab0a836c5446ddb66e46c6a87ebe96
2017-06-13 22:18:35,259 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/0fab0a836c5446ddb66e46c6a87ebe96, entries=299608, sequenceid=36742865, filesize=14.3 M
2017-06-13 22:18:35,260 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=11.27 MB/11821880 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 935ms, sequenceid=36742865, compaction requested=true
2017-06-13 22:20:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.79 GB, freeSize=191.87 MB, max=3.98 GB, blockCount=61299, accesses=226025855, hits=115745566, hitRatio=51.21%, , cachingAccesses=222477061, cachingHits=115720041, cachingHitsRatio=52.01%, evictions=46211, evicted=104622998, evictedPerRun=2264.028076171875
2017-06-13 22:20:25,382 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497392203267 with entries=448, filesize=121.71 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497392425342
2017-06-13 22:20:41,007 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789., current region memstore size 128.07 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:20:41,396 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2084224, memsize=128.1 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/.tmp/602ecffc452f4a5ca60692a78f8bf79e
2017-06-13 22:20:41,414 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/info/602ecffc452f4a5ca60692a78f8bf79e, entries=37341, sequenceid=2084224, filesize=726.9 K
2017-06-13 22:20:41,415 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.07 MB/134292984, currentsize=1.27 MB/1327000 for region ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. in 408ms, sequenceid=2084224, compaction requested=false
2017-06-13 22:22:30,731 INFO  [sync.3] wal.FSHLog: Slow sync cost: 2923 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 22:22:33,682 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:22:34,561 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36743372, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/820a95dbd7f34ac6b77ccbdc2e790488
2017-06-13 22:22:34,596 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/820a95dbd7f34ac6b77ccbdc2e790488, entries=299608, sequenceid=36743372, filesize=14.3 M
2017-06-13 22:22:34,597 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=1.61 MB/1691264 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 915ms, sequenceid=36743372, compaction requested=true
2017-06-13 22:22:34,599 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 22:22:34,599 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 22:22:34,600 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=62354, currentSize=4143545832, freeSize=129952024, maxSize=4273497856, heapSize=4143545832, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:22:37,901 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into b4cd48664c5d462cb675bbeba41c4666(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 3sec to execute.
2017-06-13 22:22:37,901 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21477316917083792; duration=3sec
2017-06-13 22:22:43,847 INFO  [sync.0] wal.FSHLog: Slow sync cost: 632 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 22:25:08,134 INFO  [sync.3] wal.FSHLog: Slow sync cost: 475 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 22:25:16,370 INFO  [sync.1] wal.FSHLog: Slow sync cost: 872 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 22:25:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.81 MB, max=3.98 GB, blockCount=61076, accesses=226266165, hits=115894893, hitRatio=51.22%, , cachingAccesses=222715125, cachingHits=115869368, cachingHitsRatio=52.03%, evictions=46252, evicted=104711958, evictedPerRun=2263.944580078125
2017-06-13 22:25:25,004 INFO  [sync.3] wal.FSHLog: Slow sync cost: 368 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 22:25:25,094 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:25:26,133 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36743928, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/102b7768114a475e8afd5f2e9e831124
2017-06-13 22:25:26,182 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/102b7768114a475e8afd5f2e9e831124, entries=299608, sequenceid=36743928, filesize=14.3 M
2017-06-13 22:25:26,184 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=10.29 MB/10787080 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 1090ms, sequenceid=36743928, compaction requested=true
2017-06-13 22:27:30,614 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497392425342 with entries=430, filesize=121.80 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497392850558
2017-06-13 22:27:30,615 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497390317239 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497390317239
2017-06-13 22:27:30,635 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497390825781 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497390825781
2017-06-13 22:27:30,666 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391039371 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391039371
2017-06-13 22:27:30,748 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391065352 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391065352
2017-06-13 22:27:30,882 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391963091 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391963091
2017-06-13 22:27:34,133 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228., current region memstore size 128.72 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:27:35,027 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=267053, memsize=128.7 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/.tmp/7a0b939f903e4154b764a85346313fd9
2017-06-13 22:27:35,062 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/info/7a0b939f903e4154b764a85346313fd9, entries=658042, sequenceid=267053, filesize=6.6 M
2017-06-13 22:27:35,063 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.72 MB/134972048, currentsize=30.89 MB/32395232 for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. in 930ms, sequenceid=267053, compaction requested=true
2017-06-13 22:27:35,402 INFO  [sync.3] wal.FSHLog: Slow sync cost: 172 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK]]
2017-06-13 22:30:16,432 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 267 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK]]
2017-06-13 22:30:16,466 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497392850558 with entries=296, filesize=123.42 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393016139
2017-06-13 22:30:17,156 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:30:17,987 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36744435, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/71fdb3dfd3194dc0962ed51f98f3d794
2017-06-13 22:30:18,004 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/71fdb3dfd3194dc0962ed51f98f3d794, entries=299608, sequenceid=36744435, filesize=14.3 M
2017-06-13 22:30:18,007 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=9.37 MB/9824184 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 851ms, sequenceid=36744435, compaction requested=true
2017-06-13 22:30:18,009 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 22:30:18,009 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 22:30:18,010 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61077, currentSize=4059819392, freeSize=213678464, maxSize=4273497856, heapSize=4059819392, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:30:20,133 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into d922e0baa69745f49f83f666f4e46734(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 22:30:20,134 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21477780326545676; duration=2sec
2017-06-13 22:30:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.78 MB, max=3.98 GB, blockCount=61077, accesses=226268442, hits=115894911, hitRatio=51.22%, , cachingAccesses=222715156, cachingHits=115869386, cachingHitsRatio=52.03%, evictions=46282, evicted=104711970, evictedPerRun=2262.47705078125
2017-06-13 22:32:57,525 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:32:58,475 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36744991, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/f5e563451c5040c59ab98082b99e943c
2017-06-13 22:32:58,495 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/f5e563451c5040c59ab98082b99e943c, entries=299608, sequenceid=36744991, filesize=14.3 M
2017-06-13 22:32:58,496 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=11.27 MB/11821880 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 971ms, sequenceid=36744991, compaction requested=true
2017-06-13 22:35:17,310 INFO  [sync.4] wal.FSHLog: Slow sync cost: 442 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-8ab76f9c-ee05-4ec0-897a-8718ab89635f,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK]]
2017-06-13 22:35:24,866 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 2174 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK]]
2017-06-13 22:35:24,885 INFO  [sync.2] wal.FSHLog: Slow sync cost: 1315 ms, current pipeline: []
2017-06-13 22:35:24,885 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393016139 with entries=500, filesize=128.08 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393322636
2017-06-13 22:35:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.78 MB, max=3.98 GB, blockCount=61077, accesses=226268478, hits=115894947, hitRatio=51.22%, , cachingAccesses=222715192, cachingHits=115869422, cachingHitsRatio=52.03%, evictions=46312, evicted=104711970, evictedPerRun=2261.011474609375
2017-06-13 22:36:14,657 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228., current region memstore size 128.84 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:36:15,477 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=267172, memsize=128.8 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/.tmp/08ce1d347e604dd2af92ea0841e4dd64
2017-06-13 22:36:15,491 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/info/08ce1d347e604dd2af92ea0841e4dd64, entries=661500, sequenceid=267172, filesize=6.7 M
2017-06-13 22:36:15,492 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.84 MB/135098872, currentsize=24.25 MB/25432968 for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. in 835ms, sequenceid=267172, compaction requested=true
2017-06-13 22:36:15,494 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228.
2017-06-13 22:36:15,494 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/.tmp, totalSize=31.1 M
2017-06-13 22:36:15,503 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61077, currentSize=4059819392, freeSize=213678464, maxSize=4273497856, heapSize=4059819392, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:36:18,998 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228., current region memstore size 130.30 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:36:19,716 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. into 06e8920f216441929f76c469c37dbb97(size=21.1 M), total size for store is 306.3 M. This selection was in queue for 0sec, and took 4sec to execute.
2017-06-13 22:36:19,716 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228., storeName=info, fileCount=3, fileSize=31.1 M, priority=6, time=21478137811586092; duration=4sec
2017-06-13 22:36:19,866 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=267293, memsize=130.3 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/.tmp/328c3b60b1c94aeb9ddb9caa9376c05b
2017-06-13 22:36:19,884 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/info/328c3b60b1c94aeb9ddb9caa9376c05b, entries=667114, sequenceid=267293, filesize=6.8 M
2017-06-13 22:36:19,885 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~130.30 MB/136628816, currentsize=13.67 MB/14339032 for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. in 887ms, sequenceid=267293, compaction requested=true
2017-06-13 22:37:53,406 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:37:56,679 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 1654 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-5f5eea04-33fc-449d-8bb6-d372303a99c7,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK]]
2017-06-13 22:37:56,692 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36745498, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/f1eb6b8d438941589682c7327b230172
2017-06-13 22:37:57,187 INFO  [sync.3] wal.FSHLog: Slow sync cost: 778 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-6e0a3279-95d2-4bff-ac72-ed8e01ad47ab,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK]]
2017-06-13 22:37:57,201 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/f1eb6b8d438941589682c7327b230172, entries=299608, sequenceid=36745498, filesize=14.3 M
2017-06-13 22:37:57,207 INFO  [sync.4] wal.FSHLog: Slow sync cost: 527 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-50b6de57-9040-41d4-80e9-bc62db9d1c5c,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-6e0a3279-95d2-4bff-ac72-ed8e01ad47ab,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK]]
2017-06-13 22:37:57,233 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393322636 with entries=270, filesize=128.15 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393474993
2017-06-13 22:37:57,240 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=29.02 MB/30432536 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 3834ms, sequenceid=36745498, compaction requested=true
2017-06-13 22:37:57,242 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 22:37:57,242 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 22:37:57,243 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61077, currentSize=4059819392, freeSize=213678464, maxSize=4273497856, heapSize=4059819392, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:37:59,190 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into b0b80edda478436db03d4fbe7088f5d8(size=14.3 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 22:37:59,190 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21478239560018901; duration=1sec
2017-06-13 22:40:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.78 MB, max=3.98 GB, blockCount=61077, accesses=226274178, hits=115894947, hitRatio=51.22%, , cachingAccesses=222715192, cachingHits=115869422, cachingHitsRatio=52.03%, evictions=46342, evicted=104711970, evictedPerRun=2259.5478515625
2017-06-13 22:40:39,798 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:40:40,684 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36746054, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/248f4320e9d54656b89184f3d79db4d1
2017-06-13 22:40:40,700 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/248f4320e9d54656b89184f3d79db4d1, entries=299608, sequenceid=36746054, filesize=14.3 M
2017-06-13 22:40:40,712 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=9.57 MB/10031064 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 914ms, sequenceid=36746054, compaction requested=true
2017-06-13 22:43:41,663 INFO  [sync.0] wal.FSHLog: Slow sync cost: 1200 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-d7a093e4-6437-4935-b4a2-0decf44fabea,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152,DISK]]
2017-06-13 22:43:44,340 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393474993 with entries=459, filesize=121.88 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393824255
2017-06-13 22:45:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.78 MB, max=3.98 GB, blockCount=61077, accesses=226275720, hits=115894947, hitRatio=51.22%, , cachingAccesses=222715192, cachingHits=115869422, cachingHitsRatio=52.03%, evictions=46372, evicted=104711970, evictedPerRun=2258.086181640625
2017-06-13 22:45:58,151 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:45:59,078 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36746561, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/3f4365bfb33f4f7c878d4420df280bc8
2017-06-13 22:45:59,125 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/3f4365bfb33f4f7c878d4420df280bc8, entries=299608, sequenceid=36746561, filesize=14.3 M
2017-06-13 22:45:59,125 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=7.73 MB/8101888 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 974ms, sequenceid=36746561, compaction requested=true
2017-06-13 22:45:59,127 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 22:45:59,127 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 22:45:59,128 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61077, currentSize=4059819392, freeSize=213678464, maxSize=4273497856, heapSize=4059819392, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:46:00,336 INFO  [sync.3] wal.FSHLog: Slow sync cost: 457 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:46:07,367 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into db62b2758d174174aa7f104473d63f0b(size=14.3 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 8sec to execute.
2017-06-13 22:46:07,368 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21478721444928490; duration=8sec
2017-06-13 22:46:35,983 INFO  [sync.2] wal.FSHLog: Slow sync cost: 834 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:48:33,413 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:48:34,346 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36747117, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/4eff059aa82e462e8577287ac5668e79
2017-06-13 22:48:34,364 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/4eff059aa82e462e8577287ac5668e79, entries=299608, sequenceid=36747117, filesize=14.3 M
2017-06-13 22:48:34,365 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=11.27 MB/11821880 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 952ms, sequenceid=36747117, compaction requested=true
2017-06-13 22:48:35,728 INFO  [sync.1] wal.FSHLog: Slow sync cost: 145 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:48:51,547 INFO  [sync.2] wal.FSHLog: Slow sync cost: 1046 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:50:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.78 MB, max=3.98 GB, blockCount=61077, accesses=226277966, hits=115894947, hitRatio=51.22%, , cachingAccesses=222715192, cachingHits=115869422, cachingHitsRatio=52.03%, evictions=46402, evicted=104711970, evictedPerRun=2256.626220703125
2017-06-13 22:51:05,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. after a delay of 10657
2017-06-13 22:51:10,558 INFO  [sync.1] wal.FSHLog: Slow sync cost: 434 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:51:12,345 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393824255 with entries=447, filesize=121.96 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497394272234
2017-06-13 22:51:15,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. after a delay of 13304
2017-06-13 22:51:16,246 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c., current region memstore size 18.02 KB, and 2/2 column families' memstores are being flushed.
2017-06-13 22:51:16,407 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11125, memsize=8.4 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp/bd688e61e05f47aca39e76704b64f1ad
2017-06-13 22:51:16,527 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=11125, memsize=9.6 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/.tmp/9eca83adff1f4244b2d0be396010cc08
2017-06-13 22:51:16,571 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/ch/bd688e61e05f47aca39e76704b64f1ad, entries=43, sequenceid=11125, filesize=5.4 K
2017-06-13 22:51:16,608 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SocialMediaInsights_New/0ea2304ef4f5284c6d8e6a96c020248c/info/9eca83adff1f4244b2d0be396010cc08, entries=48, sequenceid=11125, filesize=5.7 K
2017-06-13 22:51:16,610 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~18.02 KB/18456, currentsize=0 B/0 for region SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c. in 364ms, sequenceid=11125, compaction requested=false
2017-06-13 22:54:50,833 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:54:53,032 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36747624, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/9ec9484febd74af9a313335e0b06c293
2017-06-13 22:54:53,055 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/9ec9484febd74af9a313335e0b06c293, entries=299608, sequenceid=36747624, filesize=14.3 M
2017-06-13 22:54:53,057 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=13.12 MB/13754792 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 2224ms, sequenceid=36747624, compaction requested=true
2017-06-13 22:54:53,059 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 22:54:53,059 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 22:54:53,060 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61077, currentSize=4059777376, freeSize=213720480, maxSize=4273497856, heapSize=4059777376, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 22:54:55,259 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 2de7f1ef816440f9a6ffcaa47c646824(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 22:54:55,259 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21479255376751435; duration=2sec
2017-06-13 22:54:58,152 INFO  [sync.3] wal.FSHLog: Slow sync cost: 2242 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:54:58,341 INFO  [sync.4] wal.FSHLog: Slow sync cost: 141 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:55:10,361 INFO  [sync.0] wal.FSHLog: Slow sync cost: 11976 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:55:10,361 WARN  [B.defaultRpcServer.handler=45,queue=0,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1497394498377,"responsesize":603,"method":"Multi","processingtimems":11984,"client":"hadoop3:55124","queuetimems":0,"class":"HRegionServer"}
2017-06-13 22:55:13,478 INFO  [sync.2] wal.FSHLog: Slow sync cost: 561 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:55:16,875 INFO  [sync.1] wal.FSHLog: Slow sync cost: 1462 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-751946a0-5a6f-4485-ad27-61f061359410,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 22:55:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.82 MB, max=3.98 GB, blockCount=61077, accesses=226284705, hits=115896370, hitRatio=51.22%, , cachingAccesses=222716616, cachingHits=115870844, cachingHitsRatio=52.03%, evictions=46432, evicted=104711972, evictedPerRun=2255.168212890625
2017-06-13 22:57:57,319 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497394272234 with entries=478, filesize=121.61 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497394677287
2017-06-13 22:57:57,319 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497390562635 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497390562635
2017-06-13 22:57:57,338 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391016913 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391016913
2017-06-13 22:57:57,355 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391604208 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391604208
2017-06-13 22:57:58,168 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:57:59,068 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36748180, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/863ffdd624dc45f9b30d7b4805773a2d
2017-06-13 22:57:59,089 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/863ffdd624dc45f9b30d7b4805773a2d, entries=299608, sequenceid=36748180, filesize=14.3 M
2017-06-13 22:57:59,092 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=9.57 MB/10031064 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 924ms, sequenceid=36748180, compaction requested=true
2017-06-13 22:58:05,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. after a delay of 3780
2017-06-13 22:58:09,370 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 104.09 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 22:58:10,099 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1697050, memsize=104.1 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/5a6749431ace4e77b6f338dd2d6b2bcf
2017-06-13 22:58:10,126 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/5a6749431ace4e77b6f338dd2d6b2bcf, entries=608596, sequenceid=1697050, filesize=5.8 M
2017-06-13 22:58:10,128 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~104.09 MB/109145160, currentsize=0 B/0 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 758ms, sequenceid=1697050, compaction requested=true
2017-06-13 23:00:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.82 MB, max=3.98 GB, blockCount=61077, accesses=226284705, hits=115896370, hitRatio=51.22%, , cachingAccesses=222716616, cachingHits=115870844, cachingHitsRatio=52.03%, evictions=46462, evicted=104711972, evictedPerRun=2253.7119140625
2017-06-13 23:00:56,924 INFO  [sync.3] wal.FSHLog: Slow sync cost: 4994 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:01:06,467 INFO  [sync.0] wal.FSHLog: Slow sync cost: 174 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:01:08,952 INFO  [sync.0] wal.FSHLog: Slow sync cost: 156 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:01:13,088 INFO  [sync.0] wal.FSHLog: Slow sync cost: 167 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:01:31,135 INFO  [sync.2] wal.FSHLog: Slow sync cost: 397 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:04:33,559 INFO  [sync.1] wal.FSHLog: Slow sync cost: 1015 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:04:38,272 INFO  [sync.4] wal.FSHLog: Slow sync cost: 4164 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:04:38,272 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:04:39,161 INFO  [sync.4] wal.FSHLog: Slow sync cost: 582 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:04:40,685 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36748687, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/25e8702cd7ab48ca899d0bb6225b6341
2017-06-13 23:04:40,714 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/25e8702cd7ab48ca899d0bb6225b6341, entries=299608, sequenceid=36748687, filesize=14.3 M
2017-06-13 23:04:40,717 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=13.96 MB/14643216 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 2445ms, sequenceid=36748687, compaction requested=true
2017-06-13 23:04:40,722 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 23:04:40,722 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 23:04:40,723 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61077, currentSize=4059777376, freeSize=213720480, maxSize=4273497856, heapSize=4059777376, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 23:04:47,627 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 373 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:04:57,083 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497394677287 with entries=458, filesize=122.74 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497395087220
2017-06-13 23:04:57,084 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391075593 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391075593
2017-06-13 23:04:57,169 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391087547 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391087547
2017-06-13 23:04:57,202 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497391349771 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497391349771
2017-06-13 23:04:57,254 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into 26d5a4b6c20e453ea3268f88ba58e6ea(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 16sec to execute.
2017-06-13 23:04:57,254 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21479843039665511; duration=16sec
2017-06-13 23:05:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.82 MB, max=3.98 GB, blockCount=61077, accesses=226286951, hits=115896370, hitRatio=51.22%, , cachingAccesses=222716616, cachingHits=115870844, cachingHitsRatio=52.03%, evictions=46492, evicted=104711972, evictedPerRun=2252.2578125
2017-06-13 23:05:44,438 INFO  [RS_OPEN_META-aps-hadoop4:16020-0-MetaLogRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497391544210.meta with entries=0, filesize=91 B; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497395144336.meta
2017-06-13 23:05:44,438 INFO  [RS_OPEN_META-aps-hadoop4:16020-0-MetaLogRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004..meta.1497391544210.meta to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004..meta.1497391544210.meta
2017-06-13 23:06:02,264 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., current region memstore size 130.68 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:06:03,316 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=1697161, memsize=135.7 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp/078e8e6972de4525b8e1353af328869b
2017-06-13 23:06:03,381 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/info/078e8e6972de4525b8e1353af328869b, entries=794790, sequenceid=1697161, filesize=7.4 M
2017-06-13 23:06:03,386 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~135.71 MB/142297704, currentsize=58.19 MB/61017248 for region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. in 1122ms, sequenceid=1697161, compaction requested=true
2017-06-13 23:06:03,387 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.
2017-06-13 23:06:03,387 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/SMSCampaignStatus/63c7a8f22d2c55b157db3bb51991d279/.tmp, totalSize=105.6 M
2017-06-13 23:06:03,388 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61077, currentSize=4059777376, freeSize=213720480, maxSize=4273497856, heapSize=4059777376, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 23:06:20,930 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279. into 70a39b52fafe49d2a2a07f092e86f16f(size=105.6 M), total size for store is 3.7 G. This selection was in queue for 0sec, and took 17sec to execute.
2017-06-13 23:06:20,930 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279., storeName=info, fileCount=3, fileSize=105.6 M, priority=6, time=21479925705212452; duration=17sec
2017-06-13 23:08:46,379 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:08:47,259 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36749243, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/c050ffee095d418c9ea8ed37ea399bb7
2017-06-13 23:08:47,300 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/c050ffee095d418c9ea8ed37ea399bb7, entries=299608, sequenceid=36749243, filesize=14.3 M
2017-06-13 23:08:47,301 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=8.73 MB/9157584 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 922ms, sequenceid=36749243, compaction requested=true
2017-06-13 23:10:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.88 MB, max=3.98 GB, blockCount=61076, accesses=226295725, hits=115896371, hitRatio=51.21%, , cachingAccesses=222716617, cachingHits=115870844, cachingHitsRatio=52.03%, evictions=46522, evicted=104711974, evictedPerRun=2250.8056640625
2017-06-13 23:10:33,549 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789., current region memstore size 128.51 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:10:33,962 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=2084813, memsize=128.5 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/.tmp/164df349554a433b864fe2bd3617e0cf
2017-06-13 23:10:34,046 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/info/164df349554a433b864fe2bd3617e0cf, entries=37341, sequenceid=2084813, filesize=726.9 K
2017-06-13 23:10:34,047 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.51 MB/134750712, currentsize=786.52 KB/805400 for region ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. in 498ms, sequenceid=2084813, compaction requested=true
2017-06-13 23:10:34,048 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789.
2017-06-13 23:10:34,048 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMOverview/fc702d0d5bd79bf8c74ac0c8087a3789/.tmp, totalSize=2.4 M
2017-06-13 23:10:34,050 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61076, currentSize=4059711192, freeSize=213786664, maxSize=4273497856, heapSize=4059711192, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 23:10:34,367 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 (all) file(s) in info of ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789. into 700cda6760884fc7aab9eefc958e3e63(size=1.0 M), total size for store is 1.0 M. This selection was in queue for 0sec, and took 0sec to execute.
2017-06-13 23:10:34,367 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789., storeName=info, fileCount=3, fileSize=2.4 M, priority=7, time=21480196365946023; duration=0sec
2017-06-13 23:11:44,006 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 3806 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-01d24dbb-ac60-4a67-8c97-11733f4d3f3b,DISK]]
2017-06-13 23:11:44,160 INFO  [sync.4] wal.FSHLog: Slow sync cost: 2025 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 23:11:44,173 INFO  [sync.0] wal.FSHLog: Slow sync cost: 166 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]]
2017-06-13 23:11:44,206 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497395087220 with entries=356, filesize=128.13 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497395500169
2017-06-13 23:11:44,207 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497392850558 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497392850558
2017-06-13 23:15:10,336 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:15:11,147 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36749750, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/72cdf34470ac42f98f542ba769346553
2017-06-13 23:15:11,162 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/72cdf34470ac42f98f542ba769346553, entries=299608, sequenceid=36749750, filesize=14.3 M
2017-06-13 23:15:11,163 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=9.37 MB/9824184 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 827ms, sequenceid=36749750, compaction requested=true
2017-06-13 23:15:11,165 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 23:15:11,165 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 23:15:11,166 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61076, currentSize=4059711192, freeSize=213786664, maxSize=4273497856, heapSize=4059711192, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 23:15:11,652 INFO  [sync.0] wal.FSHLog: Slow sync cost: 126 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 23:15:13,426 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into e2efa7cc1c3640b3943ac2e7352f68b1(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2017-06-13 23:15:13,426 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21480473482653420; duration=2sec
2017-06-13 23:15:23,167 INFO  [sync.3] wal.FSHLog: Slow sync cost: 8071 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 23:15:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.88 MB, max=3.98 GB, blockCount=61076, accesses=226298130, hits=115896371, hitRatio=51.21%, , cachingAccesses=222716617, cachingHits=115870844, cachingHitsRatio=52.03%, evictions=46552, evicted=104711974, evictedPerRun=2249.35498046875
2017-06-13 23:17:15,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. after a delay of 5406
2017-06-13 23:17:20,995 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf., current region memstore size 122.57 KB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:17:21,039 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=534600, memsize=122.6 K, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/.tmp/5b24aa264ea0436fa744c0edef89043a
2017-06-13 23:17:21,068 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/CampaignFactsByLinkClicksMDC/e2fdff075d0d93921fbe26ed912db4cf/info/5b24aa264ea0436fa744c0edef89043a, entries=154, sequenceid=534600, filesize=8.1 K
2017-06-13 23:17:21,069 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~122.57 KB/125512, currentsize=0 B/0 for region CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf. in 74ms, sequenceid=534600, compaction requested=false
2017-06-13 23:18:32,417 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:18:32,473 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497395500169 with entries=487, filesize=121.83 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497395912416
2017-06-13 23:18:32,478 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497392203267 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497392203267
2017-06-13 23:18:32,489 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497392425342 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497392425342
2017-06-13 23:18:32,499 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393016139 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497393016139
2017-06-13 23:18:32,510 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393474993 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497393474993
2017-06-13 23:18:32,519 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Archiving hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497393824255 to hdfs://mycluster/apps/hbase/data/oldWALs/aps-hadoop4%2C16020%2C1497285023004.default.1497393824255
2017-06-13 23:18:33,317 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36750306, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/df686af01be54fe9b23aea81de5a1051
2017-06-13 23:18:33,380 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/df686af01be54fe9b23aea81de5a1051, entries=299608, sequenceid=36750306, filesize=14.3 M
2017-06-13 23:18:33,381 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=8.73 MB/9157584 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 964ms, sequenceid=36750306, compaction requested=true
2017-06-13 23:19:07,421 ERROR [ShortCircuitCache_SlotReleaser] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x169ef04e): failed to release short-circuit shared memory slot Slot(slotIdx=75, shm=DfsClientShm(1bbe3ab8028eb960b36edb18b7cc5018)) by sending ReleaseShortCircuitAccessRequestProto to /var/lib/hadoop-hdfs/dn_socket.  Closing shared memory segment.
java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 1bbe3ab8028eb960b36edb18b7cc5018
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2017-06-13 23:19:07,421 INFO  [B.defaultRpcServer.handler=21,queue=1,port=16020] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x169ef04e): could not load 1073748751_BP-1810172115-hadoop2-1478343078462 due to InvalidToken exception.
org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /apps/hbase/data/data/default/SocialMediaAnalyticsRecipients/e67c1301dd29e91602c633432b1414f0/info/5a573afd0cb24005be66ca0d4e622228
	at org.apache.hadoop.hdfs.BlockReaderFactory.requestFileDescriptors(BlockReaderFactory.java:589)
	at org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:488)
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784)
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718)
	at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422)
	at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333)
	at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:662)
	at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:898)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:955)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1407)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:441)
	at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:910)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
	at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
	at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:217)
	at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2075)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5467)
	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2604)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2590)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2572)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2273)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:745)
2017-06-13 23:19:07,421 WARN  [ShortCircuitCache_SlotReleaser] shortcircuit.DfsClientShmManager: EndpointShmManager(DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], parent=ShortCircuitShmManager(02ab7164)): error shutting down shm: got IOException calling shutdown(SHUT_RDWR)
java.nio.channels.ClosedChannelException
	at org.apache.hadoop.util.CloseableReferenceCount.reference(CloseableReferenceCount.java:57)
	at org.apache.hadoop.net.unix.DomainSocket.shutdown(DomainSocket.java:387)
	at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.shutdown(DfsClientShmManager.java:379)
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:223)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2017-06-13 23:19:07,421 INFO  [B.defaultRpcServer.handler=21,queue=1,port=16020] hdfs.DFSClient: Access token was invalid when connecting to /hadoop4:50010 : org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /apps/hbase/data/data/default/SocialMediaAnalyticsRecipients/e67c1301dd29e91602c633432b1414f0/info/5a573afd0cb24005be66ca0d4e622228
2017-06-13 23:20:22,741 INFO  [sync.2] wal.FSHLog: Slow sync cost: 894 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 23:20:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.88 MB, max=3.98 GB, blockCount=61076, accesses=226298131, hits=115896371, hitRatio=51.21%, , cachingAccesses=222716617, cachingHits=115870844, cachingHitsRatio=52.03%, evictions=46582, evicted=104711974, evictedPerRun=2247.906494140625
2017-06-13 23:20:29,113 INFO  [sync.2] wal.FSHLog: Slow sync cost: 2320 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 23:21:24,661 INFO  [sync.3] wal.FSHLog: Slow sync cost: 2442 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-5c91e4fa-90df-4f20-8217-049e5d671c16,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-50fbd188-eca8-404b-9a7a-a85a07a1a66b,DISK]]
2017-06-13 23:22:55,872 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.58 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:22:56,745 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36750813, memsize=128.6 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/5dc9d245c72a4acc87f7d482e7e410eb
2017-06-13 23:22:56,757 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/5dc9d245c72a4acc87f7d482e7e410eb, entries=299608, sequenceid=36750813, filesize=14.3 M
2017-06-13 23:22:56,758 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.58 MB/134826720, currentsize=12.28 MB/12880656 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 886ms, sequenceid=36750813, compaction requested=true
2017-06-13 23:22:56,760 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 23:22:56,760 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp, totalSize=42.8 M
2017-06-13 23:22:56,761 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61076, currentSize=4059711192, freeSize=213786664, maxSize=4273497856, heapSize=4059711192, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 23:22:58,733 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. into d7ca3d43c8304b3484a629666e7aaa5d(size=14.2 M), total size for store is 189.2 M. This selection was in queue for 0sec, and took 1sec to execute.
2017-06-13 23:22:58,734 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., storeName=info, fileCount=3, fileSize=42.8 M, priority=6, time=21480939077836498; duration=1sec
2017-06-13 23:24:19,563 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Rolled WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497395912416 with entries=446, filesize=121.92 MB; new WAL /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497396259507
2017-06-13 23:25:23,269 INFO  [sync.1] wal.FSHLog: Slow sync cost: 2924 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 23:25:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.88 MB, max=3.98 GB, blockCount=61076, accesses=226325078, hits=115896384, hitRatio=51.21%, , cachingAccesses=222716617, cachingHits=115870844, cachingHitsRatio=52.03%, evictions=46612, evicted=104711974, evictedPerRun=2246.459716796875
2017-06-13 23:25:33,828 INFO  [sync.3] wal.FSHLog: Slow sync cost: 205 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 23:25:44,084 INFO  [MemStoreFlusher.1] regionserver.HRegion: Started memstore flush for ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86., current region memstore size 128.01 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:25:47,776 INFO  [sync.1] wal.FSHLog: Slow sync cost: 2995 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 23:25:59,113 INFO  [MemStoreFlusher.1] regionserver.DefaultStoreFlusher: Flushed, sequenceid=36751369, memsize=128.0 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/.tmp/41385a1fd15b4448817a81901dbee05b
2017-06-13 23:26:01,369 WARN  [B.defaultRpcServer.handler=22,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396350439,"responsesize":415,"method":"Scan","processingtimems":10930,"client":"hadoop4:33324","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:26:01,369 WARN  [B.defaultRpcServer.handler=46,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396350443,"responsesize":415,"method":"Scan","processingtimems":10926,"client":"hadoop4:33350","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:26:06,032 INFO  [MemStoreFlusher.1] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/ORMDetails/3a87cda1162ad8642c25d5c39fd76e86/info/41385a1fd15b4448817a81901dbee05b, entries=299608, sequenceid=36751369, filesize=14.3 M
2017-06-13 23:26:06,036 INFO  [MemStoreFlusher.1] regionserver.HRegion: Finished memstore flush of ~128.01 MB/134232648, currentsize=23.42 MB/24555888 for region ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86. in 21952ms, sequenceid=36751369, compaction requested=true
2017-06-13 23:26:08,690 INFO  [sync.2] wal.FSHLog: Slow sync cost: 2355 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 23:27:24,761 WARN  [B.defaultRpcServer.handler=11,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396405329,"responsesize":415,"method":"Scan","processingtimems":39431,"client":"hadoop4:33324","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:28:02,321 WARN  [B.defaultRpcServer.handler=21,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396470527,"responsesize":415,"method":"Scan","processingtimems":11794,"client":"hadoop4:33350","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:28:02,321 WARN  [B.defaultRpcServer.handler=9,queue=4,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396471253,"responsesize":415,"method":"Scan","processingtimems":11068,"client":"hadoop4:33324","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:28:20,881 WARN  [B.defaultRpcServer.handler=48,queue=3,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396482323,"responsesize":415,"method":"Scan","processingtimems":18558,"client":"hadoop4:33324","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:28:20,881 WARN  [B.defaultRpcServer.handler=14,queue=4,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396482323,"responsesize":415,"method":"Scan","processingtimems":18558,"client":"hadoop4:33350","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:29:30,113 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9045968] hdfs.DFSClient: Slow ReadProcessor read fields took 42856ms (threshold=30000ms); ack: seqno: 956 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 14805736456 flag: 0 flag: 0 flag: 0, targets: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 23:29:30,114 INFO  [sync.3] wal.FSHLog: Slow sync cost: 14806 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 23:29:30,114 WARN  [B.defaultRpcServer.handler=17,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1497396555307,"responsesize":8,"method":"Multi","processingtimems":14807,"client":"hadoop3:55124","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:30:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.88 MB, max=3.98 GB, blockCount=61076, accesses=226382345, hits=115896384, hitRatio=51.19%, , cachingAccesses=222716617, cachingHits=115870844, cachingHitsRatio=52.03%, evictions=46642, evicted=104711974, evictedPerRun=2245.0146484375
2017-06-13 23:30:40,184 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9045968] hdfs.DFSClient: Slow ReadProcessor read fields took 70070ms (threshold=30000ms); ack: seqno: -2 reply: SUCCESS reply: ERROR downstreamAckTimeNanos: 0 flag: 0 flag: 1, targets: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]]
2017-06-13 23:30:40,184 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9045968] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9045968
java.io.IOException: Bad response ERROR for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9045968 from datanode DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:785)
2017-06-13 23:30:40,187 WARN  [DataStreamer for file /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497396259507 block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9045968] hdfs.DFSClient: Error Recovery for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9045968 in pipeline DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]: bad datanode DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]
2017-06-13 23:31:31,844 WARN  [B.defaultRpcServer.handler=8,queue=3,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396676503,"responsesize":415,"method":"Scan","processingtimems":15340,"client":"hadoop4:60202","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:31:31,846 WARN  [B.defaultRpcServer.handler=14,queue=4,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396672939,"responsesize":415,"method":"Scan","processingtimems":18907,"client":"hadoop4:33324","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:33:15,442 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046306] hdfs.DFSClient: Slow ReadProcessor read fields took 65065ms (threshold=30000ms); ack: seqno: -2 reply: SUCCESS reply: SUCCESS reply: ERROR downstreamAckTimeNanos: 0 flag: 0 flag: 0 flag: 1, targets: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]]
2017-06-13 23:33:15,442 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046306] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046306
java.io.IOException: Bad response ERROR for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046306 from datanode DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:785)
2017-06-13 23:33:15,443 WARN  [DataStreamer for file /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497396259507 block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046306] hdfs.DFSClient: Error Recovery for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046306 in pipeline DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]: bad datanode DatanodeInfoWithStorage[hadoop1:50010,DS-0ad77247-babc-4c36-9cd0-c04ad47e0894,DISK]
2017-06-13 23:33:27,219 WARN  [sync.0] hdfs.DFSClient: Slow waitForAckedSeqno took 55008ms (threshold=30000ms)
2017-06-13 23:33:27,219 INFO  [sync.0] wal.FSHLog: Slow sync cost: 55007 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:33:27,219 WARN  [B.defaultRpcServer.handler=14,queue=4,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1497396752190,"responsesize":603,"method":"Multi","processingtimems":55029,"client":"hadoop3:46714","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:33:27,237 INFO  [sync.1] wal.FSHLog: Slow sync cost: 24320 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:33:27,237 WARN  [B.defaultRpcServer.handler=1,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1497396782916,"responsesize":8,"method":"Multi","processingtimems":24321,"client":"hadoop3:46714","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:35:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.87 MB, max=3.98 GB, blockCount=61078, accesses=226422275, hits=115896400, hitRatio=51.19%, , cachingAccesses=222716635, cachingHits=115870860, cachingHitsRatio=52.03%, evictions=46672, evicted=104711974, evictedPerRun=2243.571533203125
2017-06-13 23:35:55,725 WARN  [B.defaultRpcServer.handler=29,queue=4,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497396918342,"responsesize":415,"method":"Scan","processingtimems":37383,"client":"hadoop4:60202","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:36:25,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. after a delay of 22531
2017-06-13 23:36:35,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. after a delay of 11888
2017-06-13 23:36:45,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore requesting flush for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. after a delay of 15299
2017-06-13 23:36:48,120 INFO  [MemStoreFlusher.0] regionserver.HRegion: Started memstore flush for WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228., current region memstore size 13.67 MB, and 1/1 column families' memstores are being flushed.
2017-06-13 23:36:49,490 INFO  [MemStoreFlusher.0] regionserver.DefaultStoreFlusher: Flushed, sequenceid=267309, memsize=13.7 M, hasBloomFilter=true, into tmp file hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/.tmp/1b38b29d320f488cb6cd1f0c50ed1952
2017-06-13 23:36:49,515 INFO  [MemStoreFlusher.0] regionserver.HStore: Added hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/info/1b38b29d320f488cb6cd1f0c50ed1952, entries=71330, sequenceid=267309, filesize=663.1 K
2017-06-13 23:36:49,524 INFO  [MemStoreFlusher.0] regionserver.HRegion: Finished memstore flush of ~13.67 MB/14339032, currentsize=0 B/0 for region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. in 1404ms, sequenceid=267309, compaction requested=true
2017-06-13 23:36:49,527 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HRegion: Starting compaction on info in region WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228.
2017-06-13 23:36:49,527 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Starting compaction of 3 file(s) in info of WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. into tmpdir=hdfs://mycluster/apps/hbase/data/data/default/WebAnalyticsUserFlow/afd7ce3459eab0aa566ddaa0362ed228/.tmp, totalSize=28.5 M
2017-06-13 23:36:49,860 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] hfile.CacheConfig: blockCache=LruBlockCache{blockCount=61078, currentSize=4059719544, freeSize=213778312, maxSize=4273497856, heapSize=4059719544, minSize=4059822848, minFactor=0.95, multiSize=2029911424, multiFactor=0.5, singleSize=1014955712, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2017-06-13 23:37:08,734 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.HStore: Completed compaction of 3 file(s) in info of WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228. into 3194db0b4ddc469e8236a78e09e1d1d5(size=28.5 M), total size for store is 313.7 M. This selection was in queue for 0sec, and took 19sec to execute.
2017-06-13 23:37:08,734 INFO  [regionserver/aps-hadoop4/hadoop4:16020-shortCompactions-1497285026726] regionserver.CompactSplitThread: Completed compaction: Request = regionName=WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228., storeName=info, fileCount=3, fileSize=28.5 M, priority=6, time=21481771845142593; duration=19sec
2017-06-13 23:37:18,075 WARN  [B.defaultRpcServer.handler=49,queue=4,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497397023695,"responsesize":415,"method":"Scan","processingtimems":14380,"client":"hadoop4:60202","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:37:40,075 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046446] hdfs.DFSClient: Slow ReadProcessor read fields took 31467ms (threshold=30000ms); ack: seqno: 1904 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 4595624974 flag: 0 flag: 0 flag: 0, targets: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:37:40,075 INFO  [sync.0] wal.FSHLog: Slow sync cost: 4595 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:37:44,355 INFO  [sync.2] wal.FSHLog: Slow sync cost: 4102 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:38:16,375 WARN  [B.defaultRpcServer.handler=27,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497397071312,"responsesize":415,"method":"Scan","processingtimems":25063,"client":"hadoop4:60202","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:39:34,815 WARN  [B.defaultRpcServer.handler=17,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497397141135,"responsesize":415,"method":"Scan","processingtimems":33680,"client":"hadoop4:60202","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:40:11,290 INFO  [sync.0] wal.FSHLog: Slow sync cost: 5688 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:40:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.87 MB, max=3.98 GB, blockCount=61078, accesses=226473813, hits=115896407, hitRatio=51.17%, , cachingAccesses=222716641, cachingHits=115870866, cachingHitsRatio=52.03%, evictions=46702, evicted=104711974, evictedPerRun=2242.13037109375
2017-06-13 23:40:56,040 WARN  [B.defaultRpcServer.handler=27,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497397238388,"responsesize":415,"method":"Scan","processingtimems":17652,"client":"hadoop4:60202","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:43:03,248 INFO  [regionserver/aps-hadoop4/hadoop4:16020.leaseChecker] regionserver.RSRpcServices: Scanner 7539 lease expired on region CampaignSMSTransaction,egzAcwHIcHv6BNA53201603171209,1479277468523.9babb6ae92d23edc32fc127985a42808.
2017-06-13 23:43:26,875 INFO  [regionserver/aps-hadoop4/hadoop4:16020.leaseChecker] regionserver.RSRpcServices: Scanner 7535 lease expired on region CampaignSMSTransaction,00zAcwCbQsF2D7FQ201512120743,1479290202747.a065e67f22dbf3f3ddd6beb4cabbf222.
2017-06-13 23:45:24,945 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.78 GB, freeSize=203.87 MB, max=3.98 GB, blockCount=61078, accesses=226511708, hits=115896423, hitRatio=51.17%, , cachingAccesses=222716657, cachingHits=115870882, cachingHitsRatio=52.03%, evictions=46732, evicted=104711974, evictedPerRun=2240.691162109375
2017-06-13 23:45:57,025 WARN  [B.defaultRpcServer.handler=44,queue=4,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497397532607,"responsesize":415,"method":"Scan","processingtimems":24418,"client":"hadoop4:33324","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:46:51,935 WARN  [B.defaultRpcServer.handler=47,queue=2,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1497397591744,"responsesize":415,"method":"Scan","processingtimems":20191,"client":"hadoop5:56238","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:47:34,514 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046446] hdfs.DFSClient: Slow ReadProcessor read fields took 70056ms (threshold=30000ms); ack: seqno: -2 reply: SUCCESS reply: ERROR downstreamAckTimeNanos: 0 flag: 0 flag: 1, targets: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:47:34,514 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046446] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046446
java.io.IOException: Bad response ERROR for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046446 from datanode DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:785)
2017-06-13 23:47:34,514 WARN  [DataStreamer for file /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497396259507 block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046446] hdfs.DFSClient: Error Recovery for block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046446 in pipeline DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]: bad datanode DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK]
2017-06-13 23:47:34,517 WARN  [DataStreamer for file /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497396259507 block BP-1810172115-hadoop2-1478343078462:blk_1082753306_9046446] hdfs.DFSClient: DataStreamer Exception
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:947)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1021)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1189)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:47:34,518 WARN  [sync.2] hdfs.DFSClient: Error while syncing
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:947)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1021)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1189)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:47:34,518 WARN  [sync.3] hdfs.DFSClient: Error while syncing
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:947)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1021)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1189)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:47:34,518 ERROR [sync.2] wal.FSHLog: Error syncing, request close of WAL
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:947)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1021)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1189)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:47:34,518 ERROR [sync.3] wal.FSHLog: Error syncing, request close of WAL
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:947)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1021)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1189)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:47:34,518 INFO  [sync.2] wal.FSHLog: Slow sync cost: 70063 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:47:34,518 INFO  [sync.3] wal.FSHLog: Slow sync cost: 35633 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:47:34,518 WARN  [B.defaultRpcServer.handler=21,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1497397618884,"responsesize":1254,"method":"Multi","processingtimems":35634,"client":"hadoop3:47394","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:47:34,542 WARN  [B.defaultRpcServer.handler=35,queue=0,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1497397584442,"responsesize":125104,"method":"Multi","processingtimems":70100,"client":"hadoop3:47394","queuetimems":0,"class":"HRegionServer"}
2017-06-13 23:47:34,621 WARN  [regionserver/aps-hadoop4/hadoop4:16020.append-pool1-t1] wal.FSHLog: Failed appending 11145, requesting roll of WAL
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:947)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1021)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1189)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:47:43,774 INFO  [regionserver/aps-hadoop4/hadoop4:16020.leaseChecker] regionserver.RSRpcServices: Scanner 7541 lease expired on region CampaignSMSTransaction,00zAcwQqNaHXCYUK201508191121,1479280046343.20baaf969d2c37ab4f7bfdd7de033e44.
2017-06-13 23:48:41,494 WARN  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] hdfs.DFSClient: Slow waitForAckedSeqno took 66948ms (threshold=30000ms)
2017-06-13 23:48:41,519 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Slow sync cost: 66974 ms, current pipeline: [DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]
2017-06-13 23:48:41,520 ERROR [regionserver/aps-hadoop4/hadoop4:16020.logRoller] wal.FSHLog: Failed close of WAL writer hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497396259507, unflushedEntries=3
org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: On sync
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1893)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:964)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:733)
	at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: On sync
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2073)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1950)
	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed appending 11145, requesting roll of WAL
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:2187)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2028)
	... 5 more
Caused by: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:947)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1021)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1189)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:48:41,522 FATAL [regionserver/aps-hadoop4/hadoop4:16020.logRoller] regionserver.HRegionServer: ABORTING region server aps-hadoop4,16020,1497285023004: Failed log close in log roller
org.apache.hadoop.hbase.regionserver.wal.FailedLogCloseException: hdfs://mycluster/apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497396259507, unflushedEntries=3
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:1014)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:733)
	at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: On sync
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1893)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:964)
	... 3 more
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: On sync
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2073)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1950)
	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Failed appending 11145, requesting roll of WAL
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:2187)
	at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:2028)
	... 5 more
Caused by: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:947)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1021)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1189)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:48:41,522 FATAL [regionserver/aps-hadoop4/hadoop4:16020.logRoller] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint]
2017-06-13 23:48:41,622 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] regionserver.HRegionServer: Dump of metrics as JSON on abort: {
  "beans" : [ {
    "name" : "java.lang:type=Memory",
    "modelerType" : "sun.management.MemoryImpl",
    "ObjectPendingFinalizationCount" : 0,
    "NonHeapMemoryUsage" : {
      "committed" : 105324544,
      "init" : 2555904,
      "max" : -1,
      "used" : 103896648
    },
    "Verbose" : true,
    "HeapMemoryUsage" : {
      "committed" : 10683744256,
      "init" : 10737418240,
      "max" : 10683744256,
      "used" : 7159431792
    },
    "ObjectName" : "java.lang:type=Memory"
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC",
    "modelerType" : "RegionServer,sub=IPC",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "aps-hadoop4",
    "queueSize" : 0,
    "numCallsInGeneralQueue" : 0,
    "numCallsInReplicationQueue" : 0,
    "numCallsInPriorityQueue" : 0,
    "numOpenConnections" : 5,
    "numActiveHandler" : 0,
    "receivedBytes" : 30083666906,
    "exceptions.RegionMovedException" : 10,
    "authenticationSuccesses" : 0,
    "authorizationFailures" : 0,
    "TotalCallTime_num_ops" : 103120222,
    "TotalCallTime_min" : 0,
    "TotalCallTime_max" : 93787,
    "TotalCallTime_mean" : 1.9971996569208317,
    "TotalCallTime_median" : 1.0,
    "TotalCallTime_75th_percentile" : 1.0,
    "TotalCallTime_95th_percentile" : 1.0,
    "TotalCallTime_99th_percentile" : 11.059999999999945,
    "exceptions.RegionTooBusyException" : 0,
    "exceptions.FailedSanityCheckException" : 0,
    "exceptions.UnknownScannerException" : 0,
    "exceptions.OutOfOrderScannerNextException" : 0,
    "exceptions" : 48,
    "ProcessCallTime_num_ops" : 103120222,
    "ProcessCallTime_min" : 0,
    "ProcessCallTime_max" : 93787,
    "ProcessCallTime_mean" : 1.9764367943273047,
    "ProcessCallTime_median" : 1.0,
    "ProcessCallTime_75th_percentile" : 1.0,
    "ProcessCallTime_95th_percentile" : 1.0,
    "ProcessCallTime_99th_percentile" : 12.0,
    "exceptions.NotServingRegionException" : 38,
    "authorizationSuccesses" : 6254,
    "sentBytes" : 3041155237761,
    "QueueCallTime_num_ops" : 103120222,
    "QueueCallTime_min" : 0,
    "QueueCallTime_max" : 247,
    "QueueCallTime_mean" : 0.020762862593527,
    "QueueCallTime_median" : 0.0,
    "QueueCallTime_75th_percentile" : 0.0,
    "QueueCallTime_95th_percentile" : 0.0,
    "QueueCallTime_99th_percentile" : 1.0,
    "authenticationFailures" : 0
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication",
    "modelerType" : "RegionServer,sub=Replication",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "aps-hadoop4",
    "sink.appliedHFiles" : 0,
    "sink.appliedOps" : 0,
    "sink.ageOfLastAppliedOp" : 0,
    "sink.appliedBatches" : 0
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server",
    "modelerType" : "RegionServer,sub=Server",
    "tag.zookeeperQuorum" : "aps-hadoop6:2181,aps-hadoop7:2181,aps-hadoop4:2181",
    "tag.serverName" : "aps-hadoop4,16020,1497285023004",
    "tag.clusterId" : "fea6e6f3-b462-4e4c-946c-f35c02a0d2f4",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "aps-hadoop4",
    "regionCount" : 120,
    "storeCount" : 174,
    "hlogFileCount" : 8,
    "hlogFileSize" : 780548355,
    "storeFileCount" : 66,
    "memStoreSize" : 231475448,
    "storeFileSize" : 38540987358,
    "regionServerStartTime" : 1497285023004,
    "totalRequestCount" : 124873796,
    "readRequestCount" : 10280883616,
    "writeRequestCount" : 21815145,
    "checkMutateFailedCount" : 0,
    "checkMutatePassedCount" : 0,
    "storeFileIndexSize" : 1062152,
    "staticIndexSize" : 174549276,
    "staticBloomSize" : 297883026,
    "mutationsWithoutWALCount" : 0,
    "mutationsWithoutWALSize" : 0,
    "percentFilesLocal" : 83,
    "percentFilesLocalSecondaryRegions" : 0,
    "splitQueueLength" : 0,
    "compactionQueueLength" : 0,
    "flushQueueLength" : 0,
    "blockCacheFreeSize" : 213778312,
    "blockCacheCount" : 61078,
    "blockCacheSize" : 4059719544,
    "blockCacheHitCount" : 115896423,
    "blockCacheHitCountPrimary" : 115896423,
    "blockCacheMissCount" : 110650423,
    "blockCacheMissCountPrimary" : 110650423,
    "blockCacheEvictionCount" : 104711974,
    "blockCacheEvictionCountPrimary" : 104711974,
    "blockCacheCountHitPercent" : 51.0,
    "blockCacheExpressHitPercent" : 52,
    "updatesBlockedTime" : 0,
    "flushedCellsCount" : 351507692,
    "compactedCellsCount" : 452558230,
    "majorCompactedCellsCount" : 828346755,
    "flushedCellsSize" : 92400565464,
    "compactedCellsSize" : 32426266891,
    "majorCompactedCellsSize" : 39270836815,
    "blockedRequestCount" : 55,
    "Mutate_num_ops" : 189929,
    "Mutate_min" : 0,
    "Mutate_max" : 93787,
    "Mutate_mean" : 33.847616740992684,
    "Mutate_median" : 13.0,
    "Mutate_75th_percentile" : 18.0,
    "Mutate_95th_percentile" : 64.64999999999986,
    "Mutate_99th_percentile" : 753.8899999999915,
    "slowAppendCount" : 0,
    "slowDeleteCount" : 0,
    "Increment_num_ops" : 0,
    "Increment_min" : 0,
    "Increment_max" : 0,
    "Increment_mean" : 0.0,
    "Increment_median" : 0.0,
    "Increment_75th_percentile" : 0.0,
    "Increment_95th_percentile" : 0.0,
    "Increment_99th_percentile" : 0.0,
    "Replay_num_ops" : 0,
    "Replay_min" : 0,
    "Replay_max" : 0,
    "Replay_mean" : 0.0,
    "Replay_median" : 0.0,
    "Replay_75th_percentile" : 0.0,
    "Replay_95th_percentile" : 0.0,
    "Replay_99th_percentile" : 0.0,
    "FlushTime_num_ops" : 771,
    "FlushTime_min" : 350,
    "FlushTime_max" : 130781,
    "FlushTime_mean" : 5015.651102464332,
    "FlushTime_median" : 992.0,
    "FlushTime_75th_percentile" : 2077.0,
    "FlushTime_95th_percentile" : 23712.799999999996,
    "FlushTime_99th_percentile" : 79054.47999999995,
    "Delete_num_ops" : 0,
    "Delete_min" : 0,
    "Delete_max" : 0,
    "Delete_mean" : 0.0,
    "Delete_median" : 0.0,
    "Delete_75th_percentile" : 0.0,
    "Delete_95th_percentile" : 0.0,
    "Delete_99th_percentile" : 0.0,
    "splitRequestCount" : 1,
    "splitSuccessCount" : 1,
    "slowGetCount" : 0,
    "Get_num_ops" : 3,
    "Get_min" : 0,
    "Get_max" : 311,
    "Get_mean" : 104.66666666666667,
    "Get_median" : 3.0,
    "Get_75th_percentile" : 311.0,
    "Get_95th_percentile" : 311.0,
    "Get_99th_percentile" : 311.0,
    "ScanNext_num_ops" : 102917099,
    "ScanNext_min" : 0,
    "ScanNext_max" : 2121511,
    "ScanNext_mean" : 29115.87865524659,
    "ScanNext_median" : 72822.5,
    "ScanNext_75th_percentile" : 93197.75,
    "ScanNext_95th_percentile" : 93351.65,
    "ScanNext_99th_percentile" : 93400.53,
    "slowPutCount" : 386,
    "slowIncrementCount" : 0,
    "Append_num_ops" : 0,
    "Append_min" : 0,
    "Append_max" : 0,
    "Append_mean" : 0.0,
    "Append_median" : 0.0,
    "Append_75th_percentile" : 0.0,
    "Append_95th_percentile" : 0.0,
    "Append_99th_percentile" : 0.0,
    "SplitTime_num_ops" : 1,
    "SplitTime_min" : 1469,
    "SplitTime_max" : 1469,
    "SplitTime_mean" : 1469.0,
    "SplitTime_median" : 1469.0,
    "SplitTime_75th_percentile" : 1469.0,
    "SplitTime_95th_percentile" : 1469.0,
    "SplitTime_99th_percentile" : 1469.0
  } ]
}
2017-06-13 23:48:41,684 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] regionserver.HRegionServer: STOPPED: Failed log close in log roller
2017-06-13 23:48:41,684 INFO  [regionserver/aps-hadoop4/hadoop4:16020.logRoller] regionserver.LogRoller: LogRoller exiting.
2017-06-13 23:48:41,684 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2017-06-13 23:48:41,684 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.HRegionServer: Stopping infoServer
2017-06-13 23:48:41,685 INFO  [SplitLogWorker-aps-hadoop4:16020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting. 
2017-06-13 23:48:41,685 INFO  [SplitLogWorker-aps-hadoop4:16020] regionserver.SplitLogWorker: SplitLogWorker aps-hadoop4,16020,1497285023004 exiting
2017-06-13 23:48:41,686 INFO  [regionserver/aps-hadoop4/hadoop4:16020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16030
2017-06-13 23:48:41,787 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.HeapMemoryManager: Stoping HeapMemoryTuner chore.
2017-06-13 23:48:41,787 INFO  [regionserver/aps-hadoop4/hadoop4:16020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2017-06-13 23:48:41,787 INFO  [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2017-06-13 23:48:41,787 INFO  [regionserver/aps-hadoop4/hadoop4:16020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2017-06-13 23:48:41,787 INFO  [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2017-06-13 23:48:41,787 INFO  [StoreCloserThread-UserAnalytics360UserReports,66666666,1478428457233.063d297f1b531d249ef34545ce71c266.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,787 INFO  [StoreCloserThread-HsearchIndexConfig,,1478428288064.c202098b27beefc3fb20a3d8fbbd4130.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,788 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed UserAnalytics360UserReports,66666666,1478428457233.063d297f1b531d249ef34545ce71c266.
2017-06-13 23:48:41,788 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed HsearchIndexConfig,,1478428288064.c202098b27beefc3fb20a3d8fbbd4130.
2017-06-13 23:48:41,788 INFO  [StoreCloserThread-WorkflowJobStatus,33333333,1478428303673.f59ea6027eed4692224ee7fd19bd0721.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,788 INFO  [StoreCloserThread-CampaignSummaryDetailGeographyByCity,,1478428044795.0d7dc768580bf53b4e5310a049090e50.-1] regionserver.HStore: Closed Geo
2017-06-13 23:48:41,788 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed WorkflowJobStatus,33333333,1478428303673.f59ea6027eed4692224ee7fd19bd0721.
2017-06-13 23:48:41,788 INFO  [StoreCloserThread-CampaignSummaryDetailGeographyByCity,,1478428044795.0d7dc768580bf53b4e5310a049090e50.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,788 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignSummaryDetailGeographyByCity,,1478428044795.0d7dc768580bf53b4e5310a049090e50.
2017-06-13 23:48:41,788 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.HRegionServer: aborting server aps-hadoop4,16020,1497285023004
2017-06-13 23:48:41,788 INFO  [StoreCloserThread-CampaignSummaryOverview,66666666,1478427980057.4d7b4fef344b1f1af629909a194477a7.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,788 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignSummaryOverview,66666666,1478427980057.4d7b4fef344b1f1af629909a194477a7.
2017-06-13 23:48:41,788 INFO  [regionserver/aps-hadoop4/hadoop4:16020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15c3e07cc6f923f
2017-06-13 23:48:41,791 INFO  [regionserver/aps-hadoop4/hadoop4:16020] zookeeper.ZooKeeper: Session: 0x15c3e07cc6f923f closed
2017-06-13 23:48:41,791 INFO  [regionserver/aps-hadoop4/hadoop4:16020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-06-13 23:48:41,791 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.HRegionServer: Waiting on 115 regions to close
2017-06-13 23:48:41,796 INFO  [StoreCloserThread-SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,796 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed SMSCampaignStatus,zAcweT3AfSTSKL,1489129022632.68378a5fea855a228a5514a2cc696500.
2017-06-13 23:48:41,796 INFO  [StoreCloserThread-UserAnalytics360OverviewChannels,33333333,1478428300551.370362b5f1102029ab61e1c12cb551df.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,797 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed UserAnalytics360OverviewChannels,33333333,1478428300551.370362b5f1102029ab61e1c12cb551df.
2017-06-13 23:48:41,798 INFO  [StoreCloserThread-SMSCampaignStatus,zAcw04QY9Q33SC,1493708419209.519332ebbcfca59b565641cd7caa2cda.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,798 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed SMSCampaignStatus,zAcw04QY9Q33SC,1493708419209.519332ebbcfca59b565641cd7caa2cda.
2017-06-13 23:48:41,798 INFO  [StoreCloserThread-WebAnalyticsUserFlow_Old,66666666,1478513262242.15a49ea7ef27eda43230cbae2aa52d23.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,798 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed WebAnalyticsUserFlow_Old,66666666,1478513262242.15a49ea7ef27eda43230cbae2aa52d23.
2017-06-13 23:48:41,802 INFO  [StoreCloserThread-CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,802 ERROR [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Memstore size is 34056
2017-06-13 23:48:41,802 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignFactsByLinkClicksMDC,cccccccc,1478428072397.e2fdff075d0d93921fbe26ed912db4cf.
2017-06-13 23:48:41,802 INFO  [StoreCloserThread-SocialMediaPages,33333333,1478427791732.aec6031e51fae21dc99a4286014066f6.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,802 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed SocialMediaPages,33333333,1478427791732.aec6031e51fae21dc99a4286014066f6.
2017-06-13 23:48:41,802 INFO  [StoreCloserThread-SocialMediaPages,66666666,1478427791732.c96590e78a22c42f6f6d2662407c9c9a.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,802 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed SocialMediaPages,66666666,1478427791732.c96590e78a22c42f6f6d2662407c9c9a.
2017-06-13 23:48:41,802 INFO  [StoreCloserThread-CampaignSummaryFactsByEMailClient,cccccccc,1478428266204.66b046e29ccf739427e1d6751d426918.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,802 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignSummaryFactsByEMailClient,cccccccc,1478428266204.66b046e29ccf739427e1d6751d426918.
2017-06-13 23:48:41,803 INFO  [StoreCloserThread-WebAnalyticsUserFlow,cDGwVW7RALanguageEnglish01493633934,1497288233889.bd9dd4b2d0aec2181759670c9fc4ec2b.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,803 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed WebAnalyticsUserFlow,cDGwVW7RALanguageEnglish01493633934,1497288233889.bd9dd4b2d0aec2181759670c9fc4ec2b.
2017-06-13 23:48:41,805 INFO  [StoreCloserThread-SocialMediaInsights_New_Bkp,99999999,1478427815748.95d7889e6610efe03b15db464cd990e0.-1] regionserver.HStore: Closed ch
2017-06-13 23:48:41,805 INFO  [StoreCloserThread-WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,805 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed WebAnalyticsUserFlow,cDGwXYiRALanguageEnglish11496387276,1497288233889.afd7ce3459eab0aa566ddaa0362ed228.
2017-06-13 23:48:41,805 INFO  [StoreCloserThread-DashboardChannelWisePerformance,33333333,1478428370167.4d873483c34240b7afdfd1e39e3c3d87.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,805 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed DashboardChannelWisePerformance,33333333,1478428370167.4d873483c34240b7afdfd1e39e3c3d87.
2017-06-13 23:48:41,806 INFO  [StoreCloserThread-SocialMediaInsights_New_Bkp,99999999,1478427815748.95d7889e6610efe03b15db464cd990e0.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,806 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed SocialMediaInsights_New_Bkp,99999999,1478427815748.95d7889e6610efe03b15db464cd990e0.
2017-06-13 23:48:41,806 INFO  [StoreCloserThread-WebAnalyticsUserFlow_Old,33333333,1478513262242.4a3642f2de64cfaf1879da87dd5e826f.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,806 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed WebAnalyticsUserFlow_Old,33333333,1478513262242.4a3642f2de64cfaf1879da87dd5e826f.
2017-06-13 23:48:41,806 INFO  [StoreCloserThread-CampaignMailBounce,b8955bcf_a1ea_4434_8919_c75472d06b44_000_5e4a129b-77cb-493f-abf6-5f06c32ccc12_c5d62550-58e9-11e4-89b2-06ab9bb94413_3a23d9ab-c334-4fc4-a91d-1ac9ee39ab78_20150610102126,1489993247729.17564b5b0f3f5c903c620950814a3a9c.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,806 INFO  [StoreCloserThread-DashboardSummary_testing,,1492429917459.4a97df8e41ddfa906a1c68ac6774064f.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,807 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignMailBounce,b8955bcf_a1ea_4434_8919_c75472d06b44_000_5e4a129b-77cb-493f-abf6-5f06c32ccc12_c5d62550-58e9-11e4-89b2-06ab9bb94413_3a23d9ab-c334-4fc4-a91d-1ac9ee39ab78_20150610102126,1489993247729.17564b5b0f3f5c903c620950814a3a9c.
2017-06-13 23:48:41,807 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed DashboardSummary_testing,,1492429917459.4a97df8e41ddfa906a1c68ac6774064f.
2017-06-13 23:48:41,808 INFO  [StoreCloserThread-BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.-1] regionserver.HStore: Closed Em
2017-06-13 23:48:41,808 INFO  [StoreCloserThread-ORMDetails,GkMwabF20152rs_7beb98bba079171e94dc0680b817a4adD35BlogsNegativeen20170213080100,1488445267606.dca70295018398d80f18e5d670394890.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,808 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed ORMDetails,GkMwabF20152rs_7beb98bba079171e94dc0680b817a4adD35BlogsNegativeen20170213080100,1488445267606.dca70295018398d80f18e5d670394890.
2017-06-13 23:48:41,808 INFO  [StoreCloserThread-DashboardChannelWisePerformance,66666666,1478428370167.9b6b8657b0f08db16a03bfdd7405d133.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,808 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed DashboardChannelWisePerformance,66666666,1478428370167.9b6b8657b0f08db16a03bfdd7405d133.
2017-06-13 23:48:41,808 INFO  [StoreCloserThread-CampaignRecommendation,99999999,1478428313095.17d15e16dbfb0d2e73d70cb5795bf7b4.-1] regionserver.HStore: Closed Em
2017-06-13 23:48:41,808 INFO  [StoreCloserThread-CampaignRecommendation,99999999,1478428313095.17d15e16dbfb0d2e73d70cb5795bf7b4.-1] regionserver.HStore: Closed Mb
2017-06-13 23:48:41,808 INFO  [StoreCloserThread-CampaignRecommendation,99999999,1478428313095.17d15e16dbfb0d2e73d70cb5795bf7b4.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,808 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignRecommendation,99999999,1478428313095.17d15e16dbfb0d2e73d70cb5795bf7b4.
2017-06-13 23:48:41,809 INFO  [StoreCloserThread-BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.-1] regionserver.HStore: Closed Fb
2017-06-13 23:48:41,809 INFO  [StoreCloserThread-BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.-1] regionserver.HStore: Closed Fp
2017-06-13 23:48:41,809 INFO  [StoreCloserThread-BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.-1] regionserver.HStore: Closed Lnk
2017-06-13 23:48:41,809 INFO  [StoreCloserThread-test1,cccccccc,1480081852792.de85cbdc154401a72f9fed6614ba81c8.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,809 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed test1,cccccccc,1480081852792.de85cbdc154401a72f9fed6614ba81c8.
2017-06-13 23:48:41,809 INFO  [StoreCloserThread-BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.-1] regionserver.HStore: Closed Mob
2017-06-13 23:48:41,810 ERROR [ShortCircuitCache_SlotReleaser] shortcircuit.ShortCircuitCache: ShortCircuitCache(0x169ef04e): failed to release short-circuit shared memory slot Slot(slotIdx=73, shm=DfsClientShm(1bbe3ab8028eb960b36edb18b7cc5018)) by sending ReleaseShortCircuitAccessRequestProto to /var/lib/hadoop-hdfs/dn_socket.  Closing shared memory segment.
java.io.IOException: ERROR_INVALID: there is no shared memory segment registered with shmId 1bbe3ab8028eb960b36edb18b7cc5018
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:208)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2017-06-13 23:48:41,810 WARN  [ShortCircuitCache_SlotReleaser] shortcircuit.DfsClientShmManager: EndpointShmManager(DatanodeInfoWithStorage[hadoop4:50010,DS-1940f918-25a9-4136-8034-e6fd0972e5a2,DISK], parent=ShortCircuitShmManager(02ab7164)): error shutting down shm: got IOException calling shutdown(SHUT_RDWR)
java.nio.channels.ClosedChannelException
	at org.apache.hadoop.util.CloseableReferenceCount.reference(CloseableReferenceCount.java:57)
	at org.apache.hadoop.net.unix.DomainSocket.shutdown(DomainSocket.java:387)
	at org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.shutdown(DfsClientShmManager.java:379)
	at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache$SlotReleaser.run(ShortCircuitCache.java:223)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2017-06-13 23:48:41,810 INFO  [StoreCloserThread-BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.-1] regionserver.HStore: Closed QR
2017-06-13 23:48:41,810 INFO  [StoreCloserThread-BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.-1] regionserver.HStore: Closed Tw
2017-06-13 23:48:41,810 INFO  [StoreCloserThread-BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.-1] regionserver.HStore: Closed Yt
2017-06-13 23:48:41,810 INFO  [StoreCloserThread-CampaignSMSTransaction,egzAcwHIcHv6BNA53201603171209,1479277468523.9babb6ae92d23edc32fc127985a42808.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,811 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSMSTransaction,egzAcwHIcHv6BNA53201603171209,1479277468523.9babb6ae92d23edc32fc127985a42808.
2017-06-13 23:48:41,811 INFO  [StoreCloserThread-UserAnalytics360UserCampaignReports,66666666,1478428512513.2ba8a19071900346cf7108a410454a79.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,811 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed UserAnalytics360UserCampaignReports,66666666,1478428512513.2ba8a19071900346cf7108a410454a79.
2017-06-13 23:48:41,811 INFO  [StoreCloserThread-CampaignSummaryImpact,99999999,1478427995538.4b6593435f97c30880b807a906b9e526.-1] regionserver.HStore: Closed Dg
2017-06-13 23:48:41,811 INFO  [StoreCloserThread-CampaignSummaryImpact,99999999,1478427995538.4b6593435f97c30880b807a906b9e526.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,811 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSummaryImpact,99999999,1478427995538.4b6593435f97c30880b807a906b9e526.
2017-06-13 23:48:41,811 INFO  [StoreCloserThread-Recipients,,1478427584994.8c28d2a564d72b729ef4fdb3d3a39424.-1] regionserver.HStore: Closed Prop
2017-06-13 23:48:41,811 INFO  [StoreCloserThread-Recipients,,1478427584994.8c28d2a564d72b729ef4fdb3d3a39424.-1] regionserver.HStore: Closed Seg
2017-06-13 23:48:41,811 INFO  [StoreCloserThread-BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,811 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed BenchMark,INANA2712232Female4055,1479074875620.371b2d0fbfc3b5de25693fb45ed59d37.
2017-06-13 23:48:41,811 INFO  [StoreCloserThread-UserAnalytics360OverviewChannels,66666666,1478428300551.28675a2059efcd20340c50a62de77704.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,811 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed UserAnalytics360OverviewChannels,66666666,1478428300551.28675a2059efcd20340c50a62de77704.
2017-06-13 23:48:41,812 INFO  [StoreCloserThread-Recipients,,1478427584994.8c28d2a564d72b729ef4fdb3d3a39424.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,812 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed Recipients,,1478427584994.8c28d2a564d72b729ef4fdb3d3a39424.
2017-06-13 23:48:41,812 INFO  [StoreCloserThread-RecommendationToleranceValues,99999999,1478428275574.145b1836930567e057b99515086d871d.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,812 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed RecommendationToleranceValues,99999999,1478428275574.145b1836930567e057b99515086d871d.
2017-06-13 23:48:41,812 INFO  [StoreCloserThread-CampaignSummaryTrends_New,34nwldhMobileH2153208,1486102387783.a0652debed1258c1fd31f2b27b561b44.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,812 INFO  [StoreCloserThread-CampaignSummaryDetailGeographyByCity,66666666,1478428044795.4fbff9f2371e00c26137f17c526c653c.-1] regionserver.HStore: Closed Geo
2017-06-13 23:48:41,812 INFO  [StoreCloserThread-CampaignSummaryDetailGeographyByCity,66666666,1478428044795.4fbff9f2371e00c26137f17c526c653c.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,812 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignSummaryTrends_New,34nwldhMobileH2153208,1486102387783.a0652debed1258c1fd31f2b27b561b44.
2017-06-13 23:48:41,812 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSummaryDetailGeographyByCity,66666666,1478428044795.4fbff9f2371e00c26137f17c526c653c.
2017-06-13 23:48:41,812 INFO  [StoreCloserThread-DashboardCampaignPerformance,99999999,1478428360041.76c3366190b1d1f9cf082862c2515eb2.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,812 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed DashboardCampaignPerformance,99999999,1478428360041.76c3366190b1d1f9cf082862c2515eb2.
2017-06-13 23:48:41,812 INFO  [StoreCloserThread-UserAnalytics360OverviewBehaviour,,1478428297431.3c494ecf6df8e89a8e14ab2339c3fd5e.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,812 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed UserAnalytics360OverviewBehaviour,,1478428297431.3c494ecf6df8e89a8e14ab2339c3fd5e.
2017-06-13 23:48:41,812 INFO  [StoreCloserThread-HsearchIndexConfig,33333333,1478428288064.78ea07d7ee21ad26fb1da5658851d311.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,813 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed HsearchIndexConfig,33333333,1478428288064.78ea07d7ee21ad26fb1da5658851d311.
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-SocialMediaInsights_New_Bkp,,1478427815748.82a2d3dfe4a6995d4f71d40c206350f4.-1] regionserver.HStore: Closed ch
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-SocialMediaInsights_New_Bkp,,1478427815748.82a2d3dfe4a6995d4f71d40c206350f4.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-CampaignSummaryGeographyByCity,66666666,1478428019768.a182c014ade9845809df38586b1a291d.-1] regionserver.HStore: Closed Geo
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-CampaignSummaryGeographyByCity,66666666,1478428019768.a182c014ade9845809df38586b1a291d.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,813 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed SocialMediaInsights_New_Bkp,,1478427815748.82a2d3dfe4a6995d4f71d40c206350f4.
2017-06-13 23:48:41,813 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignSummaryGeographyByCity,66666666,1478428019768.a182c014ade9845809df38586b1a291d.
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-CampaignSummaryIndustrySegment,cccccccc,1478428096368.7dbc8f8ab660b7e1521dcecfb8109b16.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-SocialMediaUserBehaviour,66666666,1478427797767.8f8da615dc09c5e396aae602320a75d7.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,813 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSummaryIndustrySegment,cccccccc,1478428096368.7dbc8f8ab660b7e1521dcecfb8109b16.
2017-06-13 23:48:41,813 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed SocialMediaUserBehaviour,66666666,1478427797767.8f8da615dc09c5e396aae602320a75d7.
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-CampaignBlastScheduleTransactionCount,99999999,1478428104476.58d98ef45cf3925b2339b65d15e35166.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,813 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignBlastScheduleTransactionCount,99999999,1478428104476.58d98ef45cf3925b2339b65d15e35166.
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-DashboardCampaignLeadsChannelWise,cccccccc,1478428306794.17da2404716950425deff2bfa3e13daf.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,813 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed DashboardCampaignLeadsChannelWise,cccccccc,1478428306794.17da2404716950425deff2bfa3e13daf.
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-DashboardSummary_testing,99999999,1492429917459.9bc34fd47c1c4a6da84ae4fa4b6a4bc8.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,813 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed DashboardSummary_testing,99999999,1492429917459.9bc34fd47c1c4a6da84ae4fa4b6a4bc8.
2017-06-13 23:48:41,813 INFO  [StoreCloserThread-CampaignBlastScheduleTransactionCount,33333333,1478428104476.33dfe73d532a505da5cc0b20e72e936b.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,814 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignBlastScheduleTransactionCount,33333333,1478428104476.33dfe73d532a505da5cc0b20e72e936b.
2017-06-13 23:48:41,814 INFO  [StoreCloserThread-UserAnalytics360RecipientCampaign,66666666,1478428494703.3471642746ae08a3d2e0eea4ed66b3ee.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,814 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed UserAnalytics360RecipientCampaign,66666666,1478428494703.3471642746ae08a3d2e0eea4ed66b3ee.
2017-06-13 23:48:41,814 INFO  [StoreCloserThread-SocialMediaPages,99999999,1478427791732.1b0bd3aac2a80558aebbfc8473e6b0fe.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,814 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed SocialMediaPages,99999999,1478427791732.1b0bd3aac2a80558aebbfc8473e6b0fe.
2017-06-13 23:48:41,814 INFO  [StoreCloserThread-CampaignROIForecasting,cccccccc,1478428263066.42811392924647a14d2e401f6262c392.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,814 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignROIForecasting,cccccccc,1478428263066.42811392924647a14d2e401f6262c392.
2017-06-13 23:48:41,814 INFO  [StoreCloserThread-SocialMediaRecipients,33333333,1478427832245.bcb7ad91aa93be7fd55dad6571f7112e.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,814 INFO  [StoreCloserThread-EmailCampaignStatus,zAcw8iEjgOPENGMPW,1495939248878.f23cea5f578a7f41143aab6f340a780f.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,814 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed SocialMediaRecipients,33333333,1478427832245.bcb7ad91aa93be7fd55dad6571f7112e.
2017-06-13 23:48:41,814 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed EmailCampaignStatus,zAcw8iEjgOPENGMPW,1495939248878.f23cea5f578a7f41143aab6f340a780f.
2017-06-13 23:48:41,814 INFO  [StoreCloserThread-Recipients,33333333,1478427584994.f3317a31f7b3ff0e70b00c74f1f6e10f.-1] regionserver.HStore: Closed Prop
2017-06-13 23:48:41,814 INFO  [StoreCloserThread-Recipients,33333333,1478427584994.f3317a31f7b3ff0e70b00c74f1f6e10f.-1] regionserver.HStore: Closed Seg
2017-06-13 23:48:41,815 INFO  [StoreCloserThread-Recipients,33333333,1478427584994.f3317a31f7b3ff0e70b00c74f1f6e10f.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,815 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed Recipients,33333333,1478427584994.f3317a31f7b3ff0e70b00c74f1f6e10f.
2017-06-13 23:48:41,815 INFO  [StoreCloserThread-WeekDayList,33333333,1478427657453.28aa1bbb9aaf22b109c4e0baefe5d39f.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,815 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed WeekDayList,33333333,1478427657453.28aa1bbb9aaf22b109c4e0baefe5d39f.
2017-06-13 23:48:41,815 INFO  [StoreCloserThread-ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,816 ERROR [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Memstore size is 114242344
2017-06-13 23:48:41,816 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed ORMDetails,vwIwabF19830rs_6e099c054947ea0189c9fb2561dc9061D4newsNegativeUnited Statesen20161109205724,1484297244704.3a87cda1162ad8642c25d5c39fd76e86.
2017-06-13 23:48:41,816 INFO  [StoreCloserThread-ModelsConfigSync,cccccccc,1478428291186.fec4c70f1e27c5bde24a02183abaf346.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,816 INFO  [StoreCloserThread-CampaignSMSTransaction,FEzAcwgmpczXS99R201703231805,1493295423261.f6b6419ab6a9a7e5ef1637d7ae685413.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,816 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed ModelsConfigSync,cccccccc,1478428291186.fec4c70f1e27c5bde24a02183abaf346.
2017-06-13 23:48:41,816 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSMSTransaction,FEzAcwgmpczXS99R201703231805,1493295423261.f6b6419ab6a9a7e5ef1637d7ae685413.
2017-06-13 23:48:41,816 INFO  [StoreCloserThread-RecipientsSocialMediaInfo,,1478427803572.2fc10b953daabf4e549875f640a15c08.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,816 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed RecipientsSocialMediaInfo,,1478427803572.2fc10b953daabf4e549875f640a15c08.
2017-06-13 23:48:41,816 INFO  [StoreCloserThread-CampaignSummaryDemographics,,1478427988763.bef9ebf6ba4beace1b24ce296606defb.-1] regionserver.HStore: Closed Dg
2017-06-13 23:48:41,816 INFO  [StoreCloserThread-EmailCampaignStatus,vwIwldhdHBounceMRRH,1494838610193.60e6408b7ea9710442c29659a7115483.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,816 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed EmailCampaignStatus,vwIwldhdHBounceMRRH,1494838610193.60e6408b7ea9710442c29659a7115483.
2017-06-13 23:48:41,817 INFO  [StoreCloserThread-CampaignSummaryDemographics,,1478427988763.bef9ebf6ba4beace1b24ce296606defb.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,817 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignSummaryDemographics,,1478427988763.bef9ebf6ba4beace1b24ce296606defb.
2017-06-13 23:48:41,817 INFO  [StoreCloserThread-CampaignSMSTransaction,lozAcwnoUdDXKGD93201604141116,1479221891043.5f5c7930c81170df98b1a7c7a9204b1d.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,817 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignSMSTransaction,lozAcwnoUdDXKGD93201604141116,1479221891043.5f5c7930c81170df98b1a7c7a9204b1d.
2017-06-13 23:48:41,817 INFO  [StoreCloserThread-BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.-1] regionserver.HStore: Closed Em
2017-06-13 23:48:41,817 INFO  [StoreCloserThread-DashboardCustomerPropensity,33333333,1478428350617.f38153002996c58ea5de3814a4a2d797.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,817 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed DashboardCustomerPropensity,33333333,1478428350617.f38153002996c58ea5de3814a4a2d797.
2017-06-13 23:48:41,817 INFO  [StoreCloserThread-CampaignSummarySnapshot,66666666,1478428220782.10ae5140a97e741085cbf0c3884d6425.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,817 INFO  [StoreCloserThread-BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.-1] regionserver.HStore: Closed Fb
2017-06-13 23:48:41,817 INFO  [StoreCloserThread-SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c.-1] regionserver.HStore: Closed ch
2017-06-13 23:48:41,817 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignSummarySnapshot,66666666,1478428220782.10ae5140a97e741085cbf0c3884d6425.
2017-06-13 23:48:41,817 INFO  [StoreCloserThread-BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.-1] regionserver.HStore: Closed Fp
2017-06-13 23:48:41,817 INFO  [StoreCloserThread-BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.-1] regionserver.HStore: Closed Lnk
2017-06-13 23:48:41,818 INFO  [StoreCloserThread-BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.-1] regionserver.HStore: Closed Mob
2017-06-13 23:48:41,818 INFO  [StoreCloserThread-CampaignSMSTransaction,00zAcwCbQsF2D7FQ201512120743,1479290202747.a065e67f22dbf3f3ddd6beb4cabbf222.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,818 INFO  [StoreCloserThread-SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,818 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignSMSTransaction,00zAcwCbQsF2D7FQ201512120743,1479290202747.a065e67f22dbf3f3ddd6beb4cabbf222.
2017-06-13 23:48:41,818 ERROR [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Memstore size is 16024
2017-06-13 23:48:41,818 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed SocialMediaInsights_New,99999999,1494941509384.0ea2304ef4f5284c6d8e6a96c020248c.
2017-06-13 23:48:41,818 INFO  [StoreCloserThread-SocialMediaInsights_New,66666666,1494941509384.a736b573ad0de43a66189db5e110fd6b.-1] regionserver.HStore: Closed ch
2017-06-13 23:48:41,818 INFO  [StoreCloserThread-SocialMediaInsights_New,66666666,1494941509384.a736b573ad0de43a66189db5e110fd6b.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,818 INFO  [StoreCloserThread-BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.-1] regionserver.HStore: Closed QR
2017-06-13 23:48:41,818 INFO  [StoreCloserThread-BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.-1] regionserver.HStore: Closed Tw
2017-06-13 23:48:41,818 INFO  [StoreCloserThread-BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.-1] regionserver.HStore: Closed Yt
2017-06-13 23:48:41,818 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed SocialMediaInsights_New,66666666,1494941509384.a736b573ad0de43a66189db5e110fd6b.
2017-06-13 23:48:41,818 INFO  [StoreCloserThread-BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed BenchMark,,1479074875620.ca3b04f87378984f6ae728ce2c612634.
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-Campaigns,,1478354871976.b549cc0d22314c1b898911f0c44c8b2d.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-DashboardChannelWisePerformance,99999999,1478428370167.4b3642d80161ef691e9722dbf394fa6f.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed Campaigns,,1478354871976.b549cc0d22314c1b898911f0c44c8b2d.
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed DashboardChannelWisePerformance,99999999,1478428370167.4b3642d80161ef691e9722dbf394fa6f.
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-DashboardCampaignDetails,66666666,1478428398909.5be46e74aa03e18a031643d57acdba03.-1] regionserver.HStore: Closed DCD
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-DashboardCampaignDetails,66666666,1478428398909.5be46e74aa03e18a031643d57acdba03.-1] regionserver.HStore: Closed Lds
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-DashboardCampaignDetails,66666666,1478428398909.5be46e74aa03e18a031643d57acdba03.-1] regionserver.HStore: Closed Per
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-DashboardCampaignDetails,66666666,1478428398909.5be46e74aa03e18a031643d57acdba03.-1] regionserver.HStore: Closed Prop
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-SMSCampaignStatus,zAcwE41OAPE2K6,1489986614646.18690781354093c773e1a351396d0479.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-DashboardCampaignLeadsChannelWise,99999999,1478428306794.f6bff0ce86072056c7e9c0b81a7bd66c.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-DashboardCampaignDetails,66666666,1478428398909.5be46e74aa03e18a031643d57acdba03.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed SMSCampaignStatus,zAcwE41OAPE2K6,1489986614646.18690781354093c773e1a351396d0479.
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed DashboardCampaignDetails,66666666,1478428398909.5be46e74aa03e18a031643d57acdba03.
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed DashboardCampaignLeadsChannelWise,99999999,1478428306794.f6bff0ce86072056c7e9c0b81a7bd66c.
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-CampaignSummaryFacts,99999999,1478427939763.df9b3d430f0f5bfde0c1e2979d271e04.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-DashboardCampaignPerformance,cccccccc,1478428360041.3df79740665e77a4a3ea47fe6fbc0718.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-DashboardCustomerPropensity,99999999,1478428350617.242a78bc7c411312b32c79aac4f05688.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignSummaryFacts,99999999,1478427939763.df9b3d430f0f5bfde0c1e2979d271e04.
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed DashboardCampaignPerformance,cccccccc,1478428360041.3df79740665e77a4a3ea47fe6fbc0718.
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed DashboardCustomerPropensity,99999999,1478428350617.242a78bc7c411312b32c79aac4f05688.
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-CampaignSummaryDetailGeographyByCity,99999999,1478428044795.f6e02b69bd6d9a3603fdc286364d823b.-1] regionserver.HStore: Closed Geo
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-CampaignSummaryDetailGeographyByCity,99999999,1478428044795.f6e02b69bd6d9a3603fdc286364d823b.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [StoreCloserThread-CampaignROIForecasting,33333333,1478428263066.2c5b1325855e4fc971747f217bdd7621.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignSummaryDetailGeographyByCity,99999999,1478428044795.f6e02b69bd6d9a3603fdc286364d823b.
2017-06-13 23:48:41,819 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignROIForecasting,33333333,1478428263066.2c5b1325855e4fc971747f217bdd7621.
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-CampaignBlastScheduleTransactionByMessageID,99999999,1478428005110.9a1a6b4f35d433377c26ee708b6ab9f0.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,820 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignBlastScheduleTransactionByMessageID,99999999,1478428005110.9a1a6b4f35d433377c26ee708b6ab9f0.
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-CampaignSMSTransaction_testing,66666666,1493727222020.3c215ba34acfbfe9966c8192c4e7a320.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-RecommendationWeekday,,1478428318168.7b9a3545d56322c093e2a8251076f3d2.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,820 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignSMSTransaction_testing,66666666,1493727222020.3c215ba34acfbfe9966c8192c4e7a320.
2017-06-13 23:48:41,820 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed RecommendationWeekday,,1478428318168.7b9a3545d56322c093e2a8251076f3d2.
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-CampaignSummaryImpact,,1478427995538.fed59b8c4fe374f3ca104c505b2586c5.-1] regionserver.HStore: Closed Dg
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-CampaignSummaryImpact,,1478427995538.fed59b8c4fe374f3ca104c505b2586c5.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-DashboardAverageTimeToConversion,cccccccc,1478428382583.ac43db8941a81b2cb14aec4ab7bd4943.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-CampaignSummaryFactsByEMailClient,99999999,1478428266204.3feaf075b2fec1f7357ebdfb97d37d02.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,820 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignSummaryImpact,,1478427995538.fed59b8c4fe374f3ca104c505b2586c5.
2017-06-13 23:48:41,820 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed DashboardAverageTimeToConversion,cccccccc,1478428382583.ac43db8941a81b2cb14aec4ab7bd4943.
2017-06-13 23:48:41,820 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSummaryFactsByEMailClient,99999999,1478428266204.3feaf075b2fec1f7357ebdfb97d37d02.
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-CampaignPerformance,99999999,1478428211809.93242887b60d2de4e548bf4d53e76d37.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-SocialMediaInsightsDetail_New,66666666,1494941960807.a80f7b34f03acfb5fa33a06c8389bc3a.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,820 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignPerformance,99999999,1478428211809.93242887b60d2de4e548bf4d53e76d37.
2017-06-13 23:48:41,820 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed SocialMediaInsightsDetail_New,66666666,1494941960807.a80f7b34f03acfb5fa33a06c8389bc3a.
2017-06-13 23:48:41,820 INFO  [StoreCloserThread-SocialMediaUserBehaviour,,1478427797767.513b67d6217f4a93319844b549a12e7c.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,821 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed SocialMediaUserBehaviour,,1478427797767.513b67d6217f4a93319844b549a12e7c.
2017-06-13 23:48:41,821 INFO  [StoreCloserThread-ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,821 ERROR [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Memstore size is 56084712
2017-06-13 23:48:41,821 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed ORMOverview,99999999,1478428415029.fc702d0d5bd79bf8c74ac0c8087a3789.
2017-06-13 23:48:41,821 INFO  [StoreCloserThread-CampaignBlastScheduleTransactionByMessageID,33333333,1478428005110.03c7aa06aaebdac8debe029fd670bce5.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,821 INFO  [StoreCloserThread-SMSCampaignStatus,zAcwZHQX9BUEJD,1490800043547.0b674677ad744ffd91f7d5624480ac28.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,821 INFO  [StoreCloserThread-CampaignSMSTransaction,00zAcwQqNaHXCYUK201508191121,1479280046343.20baaf969d2c37ab4f7bfdd7de033e44.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,821 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignBlastScheduleTransactionByMessageID,33333333,1478428005110.03c7aa06aaebdac8debe029fd670bce5.
2017-06-13 23:48:41,821 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed SMSCampaignStatus,zAcwZHQX9BUEJD,1490800043547.0b674677ad744ffd91f7d5624480ac28.
2017-06-13 23:48:41,821 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSMSTransaction,00zAcwQqNaHXCYUK201508191121,1479280046343.20baaf969d2c37ab4f7bfdd7de033e44.
2017-06-13 23:48:41,821 INFO  [StoreCloserThread-ClientsBenchMark,33333333,1478428269332.2afebba1708e881b6efeb79437249bb1.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,821 INFO  [StoreCloserThread-DashboardROITrend,99999999,1478428376068.ff090358c44664a3772fa7cd629263d9.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,821 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed DashboardROITrend,99999999,1478428376068.ff090358c44664a3772fa7cd629263d9.
2017-06-13 23:48:41,821 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed ClientsBenchMark,33333333,1478428269332.2afebba1708e881b6efeb79437249bb1.
2017-06-13 23:48:41,822 INFO  [StoreCloserThread-Recipients,zAcw44PJD,1490855470855.f134dc7afaceeb7305bec7119b799530.-1] regionserver.HStore: Closed Prop
2017-06-13 23:48:41,822 INFO  [StoreCloserThread-Recipients,zAcw44PJD,1490855470855.f134dc7afaceeb7305bec7119b799530.-1] regionserver.HStore: Closed Seg
2017-06-13 23:48:41,822 INFO  [StoreCloserThread-CampaignSummaryIndustrySegment,66666666,1478428096368.6289880f7376fb3585afed107e8fb3ec.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,822 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignSummaryIndustrySegment,66666666,1478428096368.6289880f7376fb3585afed107e8fb3ec.
2017-06-13 23:48:41,822 INFO  [StoreCloserThread-EmailCampaignStatus,34nwFGkWXBounceefzTS,1490570366016.56e370443853a0cce296ed1a8c53520b.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,822 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed EmailCampaignStatus,34nwFGkWXBounceefzTS,1490570366016.56e370443853a0cce296ed1a8c53520b.
2017-06-13 23:48:41,822 INFO  [StoreCloserThread-Recipients,zAcw44PJD,1490855470855.f134dc7afaceeb7305bec7119b799530.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,822 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed Recipients,zAcw44PJD,1490855470855.f134dc7afaceeb7305bec7119b799530.
2017-06-13 23:48:41,822 INFO  [StoreCloserThread-SocialMediaRecipients,cccccccc,1478427832245.14f6bcf5177513e11f72031c6d8dc990.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,822 INFO  [StoreCloserThread-SocialMediaAnalyticsRecipients,66666666,1478427863544.e67c1301dd29e91602c633432b1414f0.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,822 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed SocialMediaRecipients,cccccccc,1478427832245.14f6bcf5177513e11f72031c6d8dc990.
2017-06-13 23:48:41,822 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed SocialMediaAnalyticsRecipients,66666666,1478427863544.e67c1301dd29e91602c633432b1414f0.
2017-06-13 23:48:41,822 INFO  [StoreCloserThread-DashboardROITrend,cccccccc,1478428376068.7bbeb8e3b953c02f5349a22c1a56eca7.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,823 INFO  [StoreCloserThread-CampaignSummaryOverview,cccccccc,1478427980057.509d7a61c7d6510b1c693f1cdf777692.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,823 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed DashboardROITrend,cccccccc,1478428376068.7bbeb8e3b953c02f5349a22c1a56eca7.
2017-06-13 23:48:41,823 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignSummaryOverview,cccccccc,1478427980057.509d7a61c7d6510b1c693f1cdf777692.
2017-06-13 23:48:41,823 INFO  [StoreCloserThread-SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,823 INFO  [StoreCloserThread-CampaignMailForwardTransaction,33333333,1478427734777.25d1071812743332488cb0d25ad7c806.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,823 ERROR [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Memstore size is 61027320
2017-06-13 23:48:41,823 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignMailForwardTransaction,33333333,1478427734777.25d1071812743332488cb0d25ad7c806.
2017-06-13 23:48:41,823 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed SMSCampaignStatus,zAcwqrugWS8L67,1488452282417.63c7a8f22d2c55b157db3bb51991d279.
2017-06-13 23:48:41,823 INFO  [StoreCloserThread-RecipientsSocialMediaInfo,99999999,1478427803572.a834062fab648fe7251a8b3f671d2ed1.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,823 INFO  [StoreCloserThread-WorkflowJobStatus,99999999,1478428303673.baae928caa2817f7d0f3c8256325b7cc.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,823 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed RecipientsSocialMediaInfo,99999999,1478427803572.a834062fab648fe7251a8b3f671d2ed1.
2017-06-13 23:48:41,823 INFO  [StoreCloserThread-CampaignSummaryFacts,66666666,1478427939763.c81813e696112c8a36b46b01c928bbdc.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,823 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed WorkflowJobStatus,99999999,1478428303673.baae928caa2817f7d0f3c8256325b7cc.
2017-06-13 23:48:41,823 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSummaryFacts,66666666,1478427939763.c81813e696112c8a36b46b01c928bbdc.
2017-06-13 23:48:41,823 INFO  [StoreCloserThread-ClientsBenchMark,66666666,1478428269332.a9e1ef22f445d57b64e82378703e0fd3.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,823 INFO  [StoreCloserThread-CampaignROIForecasting,66666666,1478428263066.de4073a2ad56e7ff8f5855d5bc1624fe.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,823 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed ClientsBenchMark,66666666,1478428269332.a9e1ef22f445d57b64e82378703e0fd3.
2017-06-13 23:48:41,823 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignROIForecasting,66666666,1478428263066.de4073a2ad56e7ff8f5855d5bc1624fe.
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Em
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Fb
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Fp
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignBlastScheduleTransaction,srzAcwp3D5XFFASRV,1492484474153.18190d106607299d031ceadd8cf40034.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Ga
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-SMSCampaignStatus,cccccccc,1493708419209.7d0daf3c7809bb67f79037b656436756.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Lnk
2017-06-13 23:48:41,824 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignBlastScheduleTransaction,srzAcwp3D5XFFASRV,1492484474153.18190d106607299d031ceadd8cf40034.
2017-06-13 23:48:41,824 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed SMSCampaignStatus,cccccccc,1493708419209.7d0daf3c7809bb67f79037b656436756.
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-IndustryBenchMark,99999999,1478428278697.0f1960ffcf9ff8e7f2fb1a0f6d9daada.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-DashboardAverageTimeToConversion,,1478428382583.55621d078219838922fe86e92d434da5.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,824 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed IndustryBenchMark,99999999,1478428278697.0f1960ffcf9ff8e7f2fb1a0f6d9daada.
2017-06-13 23:48:41,824 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed DashboardAverageTimeToConversion,,1478428382583.55621d078219838922fe86e92d434da5.
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignBlastScheduleTransactionByRecipientGUID,cccccccc,1478428471741.1ded0e08d8dad0a54f706ad7423ccb64.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Mb
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-UserAnalytics360ViralityMap,99999999,1478428479264.9fabc6c22db13e2a21b2e5974bb2688f.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,824 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignBlastScheduleTransactionByRecipientGUID,cccccccc,1478428471741.1ded0e08d8dad0a54f706ad7423ccb64.
2017-06-13 23:48:41,824 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed UserAnalytics360ViralityMap,99999999,1478428479264.9fabc6c22db13e2a21b2e5974bb2688f.
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-RecipientsSocialMediaInfo,33333333,1478427803572.8517234053d212aa7d55d117a613186f.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,824 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed RecipientsSocialMediaInfo,33333333,1478427803572.8517234053d212aa7d55d117a613186f.
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-UserAnalytics360UserCampaignReports,,1478428512513.a14e59ad7b885a742d73e75b299b3109.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,824 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed UserAnalytics360UserCampaignReports,,1478428512513.a14e59ad7b885a742d73e75b299b3109.
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Qr
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-ModelsConfigSync,,1478428291186.a697f691a8c0cd340de4a4f8123cfb98.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Ra
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Tw
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed We
2017-06-13 23:48:41,824 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed Yt
2017-06-13 23:48:41,824 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed ModelsConfigSync,,1478428291186.a697f691a8c0cd340de4a4f8123cfb98.
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-CampaignSummaryDetailGeographyByCity,cccccccc,1478428044795.7cbe4ad8f9dd7f3fb0b8a44b27b3bb03.-1] regionserver.HStore: Closed Geo
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-CampaignSummaryDetailGeographyByCity,cccccccc,1478428044795.7cbe4ad8f9dd7f3fb0b8a44b27b3bb03.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-CustomerPropensitySurrogate,cccccccc,1478427602931.5ee17b8e1d5b4ae1c22645bb40cbd812.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,825 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSummaryDetailGeographyByCity,cccccccc,1478428044795.7cbe4ad8f9dd7f3fb0b8a44b27b3bb03.
2017-06-13 23:48:41,825 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CustomerPropensitySurrogate,cccccccc,1478427602931.5ee17b8e1d5b4ae1c22645bb40cbd812.
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-UserAnalytics360OverviewBehaviour,99999999,1478428297431.6227051f8a3f958079601ad7a4b288cc.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-CampaignSummaryImpact,66666666,1478427995538.e8c08c71809b620ad30d7563a75b0dac.-1] regionserver.HStore: Closed Dg
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,825 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed UserAnalytics360OverviewBehaviour,99999999,1478428297431.6227051f8a3f958079601ad7a4b288cc.
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-CampaignSummaryImpact,66666666,1478427995538.e8c08c71809b620ad30d7563a75b0dac.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,825 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed CampaignGoalMDC,33333333,1478427913612.29d8ed7f7a9c5e6cd3537015ad67fd01.
2017-06-13 23:48:41,825 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed CampaignSummaryImpact,66666666,1478427995538.e8c08c71809b620ad30d7563a75b0dac.
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-CampaignSummaryBusinessSegment,cccccccc,1478428086929.6698745deedc2e9c642114013f4b9f9a.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,825 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-1] regionserver.HRegion: Closed CampaignSummaryBusinessSegment,cccccccc,1478428086929.6698745deedc2e9c642114013f4b9f9a.
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-WorkflowJobStatus,66666666,1478428303673.8332129f3e6fb8a8e68471935065c3fc.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,825 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-0] regionserver.HRegion: Closed WorkflowJobStatus,66666666,1478428303673.8332129f3e6fb8a8e68471935065c3fc.
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-Recipients,zAcwDSTPY,1482409859871.6f3f95e37bdc7406746f4e2a233a10fd.-1] regionserver.HStore: Closed Prop
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-Recipients,zAcwDSTPY,1482409859871.6f3f95e37bdc7406746f4e2a233a10fd.-1] regionserver.HStore: Closed Seg
2017-06-13 23:48:41,825 INFO  [StoreCloserThread-Recipients,zAcwDSTPY,1482409859871.6f3f95e37bdc7406746f4e2a233a10fd.-1] regionserver.HStore: Closed info
2017-06-13 23:48:41,826 INFO  [RS_CLOSE_REGION-aps-hadoop4:16020-2] regionserver.HRegion: Closed Recipients,zAcwDSTPY,1482409859871.6f3f95e37bdc7406746f4e2a233a10fd.
2017-06-13 23:48:41,991 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.HRegionServer: stopping server aps-hadoop4,16020,1497285023004; all regions closed.
2017-06-13 23:48:44,547 INFO  [RS_OPEN_META-aps-hadoop4:16020-0-MetaLogRoller] regionserver.LogRoller: LogRoller exiting.
2017-06-13 23:48:45,589 INFO  [aps-hadoop4,16020,1497285023004_ChoreService_1] regionserver.HRegionServer$PeriodicMemstoreFlusher: Chore: aps-hadoop4,16020,1497285023004-MemstoreFlusherChore was stopped
2017-06-13 23:48:49,197 WARN  [regionserver/aps-hadoop4/hadoop4:16020] wal.ProtobufLogWriter: Failed to write trailer, non-fatal, continuing...
java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:947)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1021)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1189)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:48:49,225 ERROR [regionserver/aps-hadoop4/hadoop4:16020] regionserver.HRegionServer: Shutdown / close of WAL failed: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]], original=[DatanodeInfoWithStorage[hadoop4:50010,DS-57010fb6-92c0-4c3e-8b9e-11233ceb7bfa,DISK], DatanodeInfoWithStorage[hadoop5:50010,DS-6a874575-5e2a-4b1d-8914-c70e465dba0e,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
2017-06-13 23:48:49,235 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.Leases: regionserver/aps-hadoop4/hadoop4:16020 closing leases
2017-06-13 23:48:49,235 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.Leases: regionserver/aps-hadoop4/hadoop4:16020 closed leases
2017-06-13 23:48:49,235 INFO  [regionserver/aps-hadoop4/hadoop4:16020] hbase.ChoreService: Chore service for: aps-hadoop4,16020,1497285023004 had [[ScheduledChore: Name: MovedRegionsCleaner for region aps-hadoop4,16020,1497285023004 Period: 120000 Unit: MILLISECONDS]] on shutdown
2017-06-13 23:48:49,235 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2017-06-13 23:48:49,235 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2017-06-13 23:48:49,235 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2017-06-13 23:48:49,235 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2017-06-13 23:48:49,247 INFO  [regionserver/aps-hadoop4/hadoop4:16020] ipc.RpcServer: Stopping server on 16020
2017-06-13 23:48:49,247 INFO  [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: stopping
2017-06-13 23:48:49,247 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2017-06-13 23:48:49,247 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2017-06-13 23:48:49,253 INFO  [regionserver/aps-hadoop4/hadoop4:16020] zookeeper.ZooKeeper: Session: 0x35c3e07cb1c95b2 closed
2017-06-13 23:48:49,253 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-06-13 23:48:49,253 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.HRegionServer: stopping server aps-hadoop4,16020,1497285023004; zookeeper connection closed.
2017-06-13 23:48:49,253 INFO  [regionserver/aps-hadoop4/hadoop4:16020] regionserver.HRegionServer: regionserver/aps-hadoop4/hadoop4:16020 exiting
2017-06-13 23:48:49,253 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
	at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2655)
2017-06-13 23:48:49,272 INFO  [pool-2-thread-1] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@7a11c4c7
2017-06-13 23:48:49,272 INFO  [pool-2-thread-1] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2017-06-13 23:48:53,132 WARN  [ResponseProcessor for block BP-1810172115-hadoop2-1478343078462:blk_1082754166_9046900] hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1810172115-hadoop2-1478343078462:blk_1082754166_9046900
java.io.EOFException: Premature EOF: no length prefix available
	at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2293)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:748)
2017-06-13 23:48:53,132 WARN  [DataStreamer for file /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497397654518 block BP-1810172115-hadoop2-1478343078462:blk_1082754166_9046900] hdfs.DFSClient: Error Recovery for block BP-1810172115-hadoop2-1478343078462:blk_1082754166_9046900 in pipeline DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK], DatanodeInfoWithStorage[hadoop6:50010,DS-08f50d3d-2a14-4033-b117-b162b0cae2ce,DISK], DatanodeInfoWithStorage[hadoop7:50010,DS-9a10c707-ebd5-4fae-8f82-b381f706fa57,DISK]: bad datanode DatanodeInfoWithStorage[hadoop4:50010,DS-9f6829c5-3834-436e-8ec2-39df06418ca4,DISK]
2017-06-13 23:48:53,143 WARN  [DataStreamer for file /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497397654518 block BP-1810172115-hadoop2-1478343078462:blk_1082754166_9046900] retry.RetryInvocationHandler: Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline over aps-hadoop2/hadoop2:8020. Not retrying because try once and fail.
org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1810172115-hadoop2-1478343078462:blk_1082754166_9046900 does not exist or is not under Constructionblk_1082754166_9046900{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-9f6829c5-3834-436e-8ec2-39df06418ca4:NORMAL:hadoop4:50010|RBW], ReplicaUC[[DISK]DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152:NORMAL:hadoop6:50010|RBW], ReplicaUC[[DISK]DS-120b90e1-5d80-447d-b8db-b8ea53661e88:NORMAL:hadoop7:50010|RBW]]}
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6457)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6525)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:911)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:963)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)

	at org.apache.hadoop.ipc.Client.call(Client.java:1455)
	at org.apache.hadoop.ipc.Client.call(Client.java:1392)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy16.updateBlockForPipeline(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:903)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy17.updateBlockForPipeline(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
	at com.sun.proxy.$Proxy18.updateBlockForPipeline(Unknown Source)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1202)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:48:53,143 WARN  [DataStreamer for file /apps/hbase/data/WALs/aps-hadoop4,16020,1497285023004/aps-hadoop4%2C16020%2C1497285023004.default.1497397654518 block BP-1810172115-hadoop2-1478343078462:blk_1082754166_9046900] hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1810172115-hadoop2-1478343078462:blk_1082754166_9046900 does not exist or is not under Constructionblk_1082754166_9046900{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-9f6829c5-3834-436e-8ec2-39df06418ca4:NORMAL:hadoop4:50010|RBW], ReplicaUC[[DISK]DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152:NORMAL:hadoop6:50010|RBW], ReplicaUC[[DISK]DS-120b90e1-5d80-447d-b8db-b8ea53661e88:NORMAL:hadoop7:50010|RBW]]}
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6457)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6525)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:911)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:963)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)

	at org.apache.hadoop.ipc.Client.call(Client.java:1455)
	at org.apache.hadoop.ipc.Client.call(Client.java:1392)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy16.updateBlockForPipeline(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:903)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy17.updateBlockForPipeline(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
	at com.sun.proxy.$Proxy18.updateBlockForPipeline(Unknown Source)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1202)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:48:53,143 ERROR [Thread-6810] hdfs.DFSClient: Failed to close inode 12251971
org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1810172115-hadoop2-1478343078462:blk_1082754166_9046900 does not exist or is not under Constructionblk_1082754166_9046900{UCState=UNDER_RECOVERY, truncateBlock=null, primaryNodeIndex=0, replicas=[ReplicaUC[[DISK]DS-9f6829c5-3834-436e-8ec2-39df06418ca4:NORMAL:hadoop4:50010|RBW], ReplicaUC[[DISK]DS-9ba4f08a-d996-4490-b27d-6c8ca9a67152:NORMAL:hadoop6:50010|RBW], ReplicaUC[[DISK]DS-120b90e1-5d80-447d-b8db-b8ea53661e88:NORMAL:hadoop7:50010|RBW]]}
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6457)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6525)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:911)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:963)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)

	at org.apache.hadoop.ipc.Client.call(Client.java:1455)
	at org.apache.hadoop.ipc.Client.call(Client.java:1392)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy16.updateBlockForPipeline(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolTranslatorPB.java:903)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy17.updateBlockForPipeline(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
	at com.sun.proxy.$Proxy18.updateBlockForPipeline(Unknown Source)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1202)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:904)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:411)
2017-06-13 23:48:53,144 INFO  [pool-2-thread-1] regionserver.ShutdownHook: Shutdown hook finished.