Hello @rajatsachan    Thanks for sharing the details into the Steps used by you to resolve the issue. This would definitely assist fellow Community Members facing similar issues. If you have no further concerns, Kindly mark the Post as Solved as well.    Thanks, Smarak ... View more

Hello @rajatsachan    Thanks for the Update concerning the post. Based on your Update, Your team used the HBCK2 Jar to move the RegionState to CLOSED. Post the concerned change, Were you able to successfully bring the Regions Online as well ? If Yes, Kindly confirm & share the steps to ensure fellow Community Users like @sainivedant41 can use the same as well.    - Smarak ... View more

Hello @rajatsachan    Based on the Master Trace shared by you, Check the RS "2385" Logs for the concerned RegionID. The RS Logs should provide additional details around the same.   - Smarak ... View more

Hello @rajatsachan    Thanks for the Update. The Screen-Shot showing Error can be ignored as they pertain to the Solr. The Regions are in CLOSING State i.e. the Regions are yet to Closed before they can be opened on a different RegionServer. The PID UI shows the Source RegionServer is "*2383*allstate*. We need to check the Master Logs & the "2383" RegionServer Logs for the Region ID & they should definitely show the reasoning for the CLOSING State.    - Smarak ... View more

Hello @rajatsachan    Thanks for the details. For the Region which you are trying to assign via HBase Shell, We need to check the Logs (Same RS A & RS B depending on the Region State in Master UI). The Logs would confirm if the Assignment is failing owing to Region failing to Close successfully on RS A or Open successfully on RS B. Depending on the Region State, Check the RS Logs & Master Logs.    Additionally, You mentioned the assignment is taking time or getting hung up. Each Command would have PID (Visible via "list_procedures" in HBase Shell or "Locks & Procedures" in HMaster UI). We can check if the PID is in RUNNABLE/WAITING/FAILED State.    Note that you have ~2K Regions per RegionServer & RegionAssignment being Slow isn't unexpected.    - Smarak ... View more

Hello @snm1523    As the Post was resolved, I am marking the Post as Solved. In future, Kindly mark the Post as Solved to ensure other Community Users can reference the Post for similar issues.   Thanks for using Cloudera Community.    - Smarak ... View more

Hello @rajatsachan    Thanks for using Cloudera Community. Based on the post, You have 2 issues: (I) Region In Transition, (II) HBCK Unavailability in HBase v2.x.    For (I), RIT indicates the Regions are likely in OPENING, FAILED_OPEN, CLOSING, FAILED_CLOSE State. For any Region to move from say, RegionServer A to RegionServer B, the Region is Closed on RegionServer A before being opened on RegionServer B. Depending on the Region State, We need to check either RegionServer A or RegionServer B Logs to identify the reasoning for the Transition. Finally, Your Environment has ~1900 Regions per RegionServer, which is way beyond the general recommendations of 300/400 Regions per RegionServer. Such large Region/RegionServer Count would cause delay in WALSplit + WALEditReplay + RegionClosing or RegionOpening etc.    For (II), KIndly review Link [1]. In short, HBase v2.x requires HBCK v2.x. You can use HBCK1 Read-Only Commands (Example: hbase hbck -details) on HBase v2.x yet any HBCK1 Fix Command won't work on HBase v2.x. You need to use the HBCK v2.x, which has to be built as described in the Link.   - Smarak   [1] https://community.cloudera.com/t5/Community-Articles/HBase-HBCK2-tool-for-HDP-3-x/ta-p/244386 ... View more

Hello @snm1523    Signal 15 tells us that something or someone intentionally killed the process. Check the Process StdOut & StdErr file for details.    - Smarak ... View more

Hello @ASIF123    Checking whether the concerned issue posted by your team has been resolved. If Yes, Kindly mark the Post as Solved. If No, Kindly review our Post dated 12/18 & share the outcome of the Action Plan shared.   - Smarak ... View more

Hello @TGH    Yes, After doing any HBCK2 Changes, Restart the Service as the Components have a Cached Version of the Metadata as well. Let us know how things goes.    - Smarak ... View more

Hello @Anks2411    Thanks for using Cloudera Community. You shall need to check the Logs of " cdh-dn-28.prod.mcs.az-eastus2.mob.nuance.com" RegionServer to confirm the reasoning for Region "8808c0e1917bf0b4acea2d83d9548463" being FAILED_CLOSE.   For any Region to be moved from RegionServer A to RegionServer B, the Region has been to closed on RegionServer A before being opened on RegionServer B. In your case, the Region is failing to close on RegionServer A & the Logs would confirm the reasoning for the same.    - Smarak ... View more