Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

*URGENT* - Cloudera 5.10 - HBASE Region Not Assigned after adding new DN

avatar
Contributor

Added 6 new DN nodes for scaling but HBASE is not assigning any regions on new DN's

Have errors like this on HBASE master - 

 

AppDispatcher,\xEB\xFDE\x08\xEB\xDD\x11\xE9\xA4I\x0AX\x0A\xF4\x8Ar,1574112884653.8808c0e1917bf0b4acea2d83d9548463. state=FAILED_CLOSE, ts=Wed Dec 16 16:53:03 UTC 2020 (3008s ago), server=cdh-dn-28.prod.mcs.az-eastus2.mob.nuance.com,60020,1608136647199
1 ACCEPTED SOLUTION

avatar
Super Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
5 REPLIES 5

avatar
Contributor

Any response on this would be highly appreciated.

avatar
Master Collaborator

You'll need to look through the Region Server log files to find the root cause of the problem. The error message you shared is not enough information to go on.

avatar
Super Collaborator

Hello @Anks2411 

 

Thanks for using Cloudera Community. You shall need to check the Logs of "cdh-dn-28.prod.mcs.az-eastus2.mob.nuance.com" RegionServer to confirm the reasoning for Region "8808c0e1917bf0b4acea2d83d9548463" being FAILED_CLOSE.

 

For any Region to be moved from RegionServer A to RegionServer B, the Region has been to closed on RegionServer A before being opened on RegionServer B. In your case, the Region is failing to close on RegionServer A & the Logs would confirm the reasoning for the same. 

 

- Smarak

avatar
Contributor

It was found to have multiple inconsistencies on hbase which were fixed yesterday. So dont see this for now. This could also be due to following message on HBASE master (this message wasn't there before repair/fix which was performed) - 

 

The Load Balancer is not enabled which will eventually cause performance degradation in HBase as Regions will not be distributed across all RegionServers. The balancer is only expected to be disabled during rolling upgrade scenarios.
 
Question - Seems this is triggered by balance_switch flag on hbase. What's the best practice for this on production environment? is it supposed to be true or false? Kindly advise.
 

avatar
Super Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login