Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher

Highlighted

Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher

Rising Star

I am running an HBase import for one (single) table.

There is one (1) HBase MasterServer and five (5) Hbase RegionServers.

One RegionServer acting up as follows:

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2015-05-18 16:47:55,901 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher$WakeupFlushThread@3eef3f55
java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:133)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:180)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:70)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:214)
        at java.lang.Thread.run(Thread.java:745)
2015-05-18 16:47:55,901 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 5000ms
2015-05-18 16:47:55,902 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 5001ms
2015-05-18 16:47:55,902 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 5001ms
2015-05-18 16:47:55,902 ERROR org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher$WakeupFlushThread@3ce91de8
java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:133)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:180)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:70)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:214)
        at java.lang.Thread.run(Thread.java:745)
2015-05-18 16:47:56,905 INFO org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher: Flushed, sequenceid=381750, memsize=738.8 M, hasBloomFilter=true, into tmp file hdfs://master:8020/hbase/data/user/clean_data/b288035311126ddbfa0303e566aa9535/.tmp/e4040aebb9e9426aa9932666418a960a
2015-05-18 16:47:56,916 INFO org.apache.hadoop.hbase.regionserver.HStore: Added hdfs://master:8020/hbase/data/user/clean_data/b288035311126ddbfa0303e566aa9535/data/e4040aebb9e9426aa9932666418a960a, entries=4109950, sequenceid=381750, filesize=20.6 M
2015-05-18 16:47:56,917 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~738.82 MB/774704008, currentsize=26.14 MB/27405632 for region user:clean_data,,1431972055397.b288035311126ddbfa0303e566aa9535. in 6672ms, sequenceid=381750, compaction requested=true
2015-05-18 16:47:56,917 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6016ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6017ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6017ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6017ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6017ms
2015-05-18 16:47:56,918 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Unblocking updates for server node3,60020,1431312134599
2015-05-18 16:47:56,918 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 6018ms

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

2 REPLIES 2

Re: Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher

Rising Star

I forgot to mention, CM reports the following warning:

"Compaction Queue Size"

"Test of whether the RegionServer's compaction queue is too full."

 

Re: Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlusher

Master Guru
Are you on CDH 5.4.0 or higher? Are you also making use of any Scans with Filters in them?
Don't have an account?
Coming from Hortonworks? Activate your account here