You can check the Hbase Master log and verify of the hbase:meta and hbase:namespace tables are online. ... View more

See if you can raise a support ticket with Cloudera. The app log needs a detailed review to know what is causing the container to get fail. ... View more

If you are still not able to run the command successfully from the Region server host, then its most likely an issue with YARN. Can you check if there are any other WARNs or ERRORs in the app log apart from log4j warnings. ... View more

can you confirm if Zookeeper service is running on this host? Apart from the log4J errors, Do you see any others in the Application log? ... View more

@Amandi Can you try to run the command from a Region server host. The connection is going to a localhost on port 2181. So if Zookeeper is not running on the host from where you are running the command, it will fail. ... View more

If the current stat in hbase:meta is OPEN, I won't suggest performing any other action to change the state in meta. I suspect there are some procedures running in the backend trying to assign the region again. Do you see any procedure as such in the "Procedure & locks" section under the Hbase Master Web UI? ... View more

If the Region server reports that the region is already OPEN, try to scan the hbase:meta table from the hbase shell and check what state that region is in. If its still in OPENING state in meta, try to change its state to OPEN. Do it for one of the region and see if that brings down the RIT count to 133. ... View more

Try to bypass the stuck procedures using the hbck2 Jar and take the Region servers for a restart. ... View more

@MrNicen firstly, You will need to check the RS log of l230-n2.<SERVER>. As per the WARn, the region is already reported as OPEN by the same RS on which its trying again to open and going into a OPENING state. It seems there are SCPs/ multiple assign procedures are running in the backend trying to open a region that is already open.  Considering that you are on a version that can neither use hbck2 jar nor the hbck1 commands,  you can give try the below : 1) Stop both the Master 2) Move the contents of MasterProcWALs to a backup location # hdfs dfs -mv /hbase/MasterProcWALs/* /tmp/ 3) Start the Masters. If the above doesn't solve the issue, I suggest raising a support case with Cloudera to review the logs. ... View more

What error are you getting while writing to HDFS? ... View more