- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Hbase regions in Transition count not decreasing
- Labels:
-
Apache Ambari
-
Apache HBase
Created 10-07-2020 01:42 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi all
i have 1 Hbase master and 3 region servers. I got a huge number (7000+) in regions in transition state.
In hbase master logs
2020-10-07 08:37:02,745 WARN [ProcExecTimeout] assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=hostname,16020,1601150311831, table=table_name, region=region_id.
for that i executed below command
assign 'region_id'
count is decreased and when i restart hbase again count increased.
so i tried hbase hbck -repair
-----------------------------------------------------------------------
NOTE: As of HBase version 2.0, the hbck tool is significantly changed.
In general, all Read-Only options are supported and can be be used
safely. Most -fix/ -repair options are NOT supported. Please see usage
below for details on which options are not supported.
i got this error.
so i need help to reduce regions in transition count or upgrade hbase in ambari to execute repair command.
Created 10-11-2020 03:39 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Manoj690
Can you check from the hmaster UI --> Procedures & Locks --> If any stuck procedure.
7K RIT is huge is this RIT is related to single table or multiple tables of region are involve. This can also happen if you less number of Region Server and huge number of regions. We generally recommend not more then ~1k Region per region server and as you have only 3 RS this limit already seems to be exhausted. Please check Region count per region server.
Created 11-01-2020 10:43 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
how to increase region count in region server.
Created 10-28-2020 10:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @Manoj690
The best way would be to check the RegionServer Logs wherein the Region are stuck in OPENING State. The fact that Count of Region-In-Transition is reducing with time & Increasing again upon HBase Service Restart indicates the Regions are likely waiting on WAL Split & Edit Replay. With 3 RegionServer, 7K RIT indicates you must be having more than 7K Regions in Total. As such, the RegionServer are overloaded & WAL Split & Edit Replay would be Slow.
There are RegionOpening Thread, which can be Increased yet the Ideal Solution would be to Reduce the RegionCount or Increase RegionServer Count to ~300-400 Region/RegionServer.
- Smarak
Created 11-22-2020 09:12 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How can i increase the region server count to ~300-400 Region/RegionServer.
Created 11-22-2020 09:28 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @Manoj690
RegionServer is a Service & your team can add the RegionServer Service interactively using via Ambari (HDP) or Cloudera Manager (CDH or CDP).
- Smarak