Hbase regions in Transition count not decreasing

[#1]

Hi all

i have 1 Hbase master and 3 region servers. I got a huge number (7000+) in regions in transition state.  

In hbase master logs 

2020-10-07 08:37:02,745 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=hostname,16020,1601150311831, table=table_name, region=region_id.

 for that i executed below command

assign 'region_id'

count is decreased and when i restart hbase again count increased.

so i tried hbase hbck -repair

-----------------------------------------------------------------------
NOTE: As of HBase version 2.0, the hbck tool is significantly changed.
In general, all Read-Only options are supported and can be be used
safely. Most -fix/ -repair options are NOT supported. Please see usage
below for details on which options are not supported.
i got this error.

so i need help to reduce regions in transition count or upgrade hbase in ambari to execute repair command.

[#2]

Hi @Manoj690 

Can you check from the hmaster UI --> Procedures &  Locks --> If any stuck procedure.

7K RIT is huge is this RIT is related to single table or multiple tables of region are involve. This can also happen if you less number of Region Server and huge number of regions. We generally recommend not more then ~1k Region per region server and as you have only 3 RS this limit already seems to be exhausted. Please check Region count per region server. 

[#3]

memstore size i reduced to 32MB. still RIT count not reduced. let me know
how to increase region count in region server.

[#4]

Hello @Manoj690 

The best way would be to check the RegionServer Logs wherein the Region are stuck in OPENING State. The fact that Count of Region-In-Transition is reducing with time & Increasing again upon HBase Service Restart indicates the Regions are likely waiting on WAL Split & Edit Replay. With 3 RegionServer, 7K RIT indicates you must be having more than 7K Regions in Total. As such, the RegionServer are overloaded & WAL Split & Edit Replay would be Slow. 

There are RegionOpening Thread, which can be Increased yet the Ideal Solution would be to Reduce the RegionCount or Increase RegionServer Count to ~300-400 Region/RegionServer. 

- Smarak

[#5]

How can i increase the region server count to ~300-400 Region/RegionServer. 

[#6]

Hello @Manoj690 

RegionServer is a Service & your team can add the RegionServer Service interactively using via Ambari (HDP) or Cloudera Manager (CDH or CDP).

- Smarak