I've got a CDH cluster with 80 datanodes/regionservers and a few HBase tables, one table in particular is very large (370TB) and gets a decent amount of writes under normal circumstances via a few MR jobs. This table had around ~12,000 regions and combined with regions from other tables, each region server had about 200 regions total.
Previously, I was on CDH 5.7.1 - a week ago went through the upgrade wizard to get to 6.0.1. On bringing HBase back online I noticed that the table described above started to split pretty quickly, with over 1000 splits in the first hour and growing quickly. I decided to let it do its thing and I'm now at 24,000 online regions with about 12,000 regions marked as "Other" (36k total).
Looking at the logs, flushes pre-upgrade were 128MB (memstore size) and flushes post-upgrade are 1-30 MB or sometimes as low as a few KB. I noticed that the function of hbase.regionserver.global.memstore.size.lower had changed from HBase 1.2 to 2.0, so moved it from 0.6 to 0.95. This hasn't seemed to have changed much.
At this point, there are anywhere from 300-500 regions per server and compaction queues have skyrocketed (region servers crashing occaisonally from high GC). The large table had over 2 million storefiles yesterday, down to 1.8 million today. My MapReduce program that writes to this table has gone from taking 5 minutes to over 10 minutes (at this point it's just stopped to allow compactions to continue).
I'm looking at off-heap writes and changing the split policy to fixed size, but I guess I have 4 questions:
Any help is appreciated.
Thanks in advance,
An HBase noob