*URGENT* - Cloudera 5.10 - HBASE Region Not Assigned after adding new DN

[#1]

Added 6 new DN nodes for scaling but HBASE is not assigning any regions on new DN's

Have errors like this on HBASE master - 

AppDispatcher,\xEB\xFDE\x08\xEB\xDD\x11\xE9\xA4I\x0AX\x0A\xF4\x8Ar,1574112884653.8808c0e1917bf0b4acea2d83d9548463. state=FAILED_CLOSE, ts=Wed Dec 16 16:53:03 UTC 2020 (3008s ago), server=cdh-dn-28.prod.mcs.az-eastus2.mob.nuance.com,60020,1608136647199

[#2]

Any response on this would be highly appreciated.

[#3]

You'll need to look through the Region Server log files to find the root cause of the problem. The error message you shared is not enough information to go on.

[#4]

Hello @Anks2411 

Thanks for using Cloudera Community. You shall need to check the Logs of "cdh-dn-28.prod.mcs.az-eastus2.mob.nuance.com" RegionServer to confirm the reasoning for Region "8808c0e1917bf0b4acea2d83d9548463" being FAILED_CLOSE.

For any Region to be moved from RegionServer A to RegionServer B, the Region has been to closed on RegionServer A before being opened on RegionServer B. In your case, the Region is failing to close on RegionServer A & the Logs would confirm the reasoning for the same. 

- Smarak

[#5]

It was found to have multiple inconsistencies on hbase which were fixed yesterday. So dont see this for now. This could also be due to following message on HBASE master (this message wasn't there before repair/fix which was performed) - 

The Load Balancer is not enabled which will eventually cause performance degradation in HBase as Regions will not be distributed across all RegionServers. The balancer is only expected to be disabled during rolling upgrade scenarios.

Question - Seems this is triggered by balance_switch flag on hbase. What's the best practice for this on production environment? is it supposed to be true or false? Kindly advise.

[#6 · Accepted solution]

Hello @Anks2411 

Thanks for sharing the Cause. To your query, Yes, HBase Balancer should be enabled & "balance_switch" should be set as "true".

Once you have no further queries, Kindly mark the Post as Solved as well. 

- Smarak