Reply
New Contributor
Posts: 8
Registered: ‎02-03-2016

HBase Master: Failed to become active master

Our table has one column family which has a TTL of 15day. So rows fall at a consistent basis. We are seeing that the no of regions are going up. Some how the regions are not getting re-used. We are currently with over 41k regions of which 19k are empty. Any pointers as to why regions are not getting reused. Our row key design is similar to what you mentioned. 2 digit salt (0 to 63) followed by hashcode of reverse time stamp in seconds, followed by a service id and finally a counter.

Our cluster is 41 nodes and we are writing rows at a rate of 4k to 6k Tps. The average row size is about 35kb.

Highlighted
Posts: 1,894
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: HBase Master: Failed to become active master

Could you illustrate your row keys further? You mention hashcode of a reversed timestamp, but what form of hash?

What does a smattering of your region row key layout look like?

HBase does store everything lexicographically sorted, so if you are seeing no reuse of regions, its bound to be your key pattern.

That said, it may be a red-herring cause you have a huge number of regions from splits already (41k). You may want to consider merging them: http://archive.cloudera.com/cdh5/cdh/5/hbase/book.html#_online_region_merges
Announcements