Reply
TS
Contributor
Posts: 84
Registered: ‎02-10-2015

Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher

I am running an HBase import for one (single) table.

There is one (1) HBase MasterServer and five (5) Hbase RegionServers.

One RegionServer acting up as follows:

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2015-05-18 16:47:55,901 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher$WakeupFlushThread@3eef3f55
java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:133)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:180)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:70)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:214)
        at java.lang.Thread.run(Thread.java:745)
2015-05-18 16:47:55,901 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 5000ms
2015-05-18 16:47:55,902 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 5001ms
2015-05-18 16:47:55,902 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 5001ms
2015-05-18 16:47:55,902 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher$WakeupFlushThread@3ce91de8
java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:133)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:180)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:70)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:214)
        at java.lang.Thread.run(Thread.java:745)
2015-05-18 16:47:56,905 INFO org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher: Flushed, sequenceid=381750, memsize=738.8 M, hasBloomFilter=true, into tmp file hdfs://master:8020/hbase/data/user/clean_data/b288035311126ddbfa0303e566aa9535/.tmp/e4040aebb9e9426aa9932666418a960a
2015-05-18 16:47:56,916 INFO org.apache.hadoop.hbase.regionserver.HStore: Added hdfs://master:8020/hbase/data/user/clean_data/b288035311126ddbfa0303e566aa9535/data/e4040aebb9e9426aa9932666418a960a, entries=4109950, sequenceid=381750, filesize=20.6 M
2015-05-18 16:47:56,917 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~738.82 MB/774704008, currentsize=26.14 MB/27405632 for region user:clean_data,,1431972055397.b288035311126ddbfa0303e566aa9535. in 6672ms, sequenceid=381750, compaction requested=true
2015-05-18 16:47:56,917 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6016ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6017ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6017ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6017ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6017ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6018ms

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

TS
Contributor
Posts: 84
Registered: ‎02-10-2015

Re: Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher

I forgot to mention, CM reports the following warning:

"Compaction Queue Size"

"Test of whether the RegionServer's compaction queue is too full."

 
Highlighted
Posts: 1,892
Kudos: 431
Solutions: 302
Registered: ‎07-31-2013

Re: Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher

Are you on CDH 5.4.0 or higher? Are you also making use of any Scans with Filters in them?
Announcements