Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hbase regions growing uncontrollably after upgrade from CDH 5.7 to CDH 6.0.1

Hbase regions growing uncontrollably after upgrade from CDH 5.7 to CDH 6.0.1

New Contributor

Hello All,

I've got a CDH cluster with 80 datanodes/regionservers and a few HBase tables, one table in particular is very large (370TB) and gets a decent amount of writes under normal circumstances via a few MR jobs. This table had around ~12,000 regions and combined with regions from other tables, each region server had about 200 regions total.

Previously, I was on CDH 5.7.1 - a week ago went through the upgrade wizard to get to 6.0.1. On bringing HBase back online I noticed that the table described above started to split pretty quickly, with over 1000 splits in the first hour and growing quickly. I decided to let it do its thing and I'm now at 24,000 online regions with about 12,000 regions marked as "Other" (36k total).

Looking at the logs, flushes pre-upgrade were 128MB (memstore size) and flushes post-upgrade are 1-30 MB or sometimes as low as a few KB. I noticed that the function of hbase.regionserver.global.memstore.size.lower had changed from HBase 1.2 to 2.0, so moved it from 0.6 to 0.95. This hasn't seemed to have changed much.

At this point, there are anywhere from 300-500 regions per server and compaction queues have skyrocketed (region servers crashing occaisonally from high GC). The large table had over 2 million storefiles yesterday, down to 1.8 million today. My MapReduce program that writes to this table has gone from taking 5 minutes to over 10 minutes (at this point it's just stopped to allow compactions to continue).

I'm looking at off-heap writes and changing the split policy to fixed size, but I guess I have 4 questions:

  1. What changed so drastically from 1.2 to 2.0 to cause all these splits after upgrade?
  2. How can I get larger flushes again?
  3. Why are there 12,000 regions marked as "Other" in my table now?
  4. Does anyone have any recommendations for what I can do to get the number of regions back under control aside from writing a script to force merges?

Any help is appreciated.

Thanks in advance,

An HBase noob

Don't have an account?
Coming from Hortonworks? Activate your account here