I'm using CDH 5.9 (hbase-1.2.0+cdh5.9.0+205) and think I've found a heap leak when creating and deleting a table.
We're running an automated test that creates and deletes an hbase table every minute (and does other things too). We noticed that when running for a long time the regionserver java heap usage trends upwards at 190MB per day. The garbage collection never quite collects all the memory and the overall trend is up.
If I disable the part of the test that creates and deletes the table, and just uses one that is alreay created then the memory usage doesn't trend up any more.
If you look at the picture, you can see where I changed the test to stop the repeated table creation - where the up trend flattens out. And then the big drop off is where I restarted the regionserver.
The regionserver java heap contains an awful lot of counter objects:
1: 7770499 1056787864 org.apache.hadoop.hbase.util.Counter$Cell
2: 7770499 186491976 [Lorg.apache.hadoop.hbase.util.Counter$Cell;
3: 7770499 186491976 org.apache.hadoop.hbase.util.Counter$Container
4: 7770498 186491952 org.apache.hadoop.hbase.util.Counter
5: 7770498 186491952 org.apache.hadoop.hbase.util.Counter$1
6: 7814877 125038032 java.util.concurrent.atomic.AtomicBoolean
7: 7777721 124443536 java.util.concurrent.atomic.AtomicReference
Any hbase experts that could check this one out and confirm?