Skip to main content
TGH
TGH
Silver ⭐️⭐️⭐️
December 8, 2020
Solved

CDH 6.3.2 - Hbase2 Table problem

I have inherited a problem.

 

48 regions in hbase:meta in transition.

 

Table has had data removed - most likely manually in an attempt to fix RIT issues. These RITs are probably a result of a network outage mid-operation.

 

Table is currently ENABLED and cannot be DISABLED (this has already been attempted by previous techie, which resulted in LOCKS/Procedures for DISABLE and DELETE as well as RITs).

 

Table is no longer required so can be deleted.  HDFS reported it as being only 6k so I removed the table directories and zapped the znodes via ZK shell.  This fixed the locks/procedures messages but CManager still reports 48 regions in transition and, as a result of this, balancing is not working.

 

What I need is a way to remove the rows from 'hbase:meta' as this is the only place where this table is still referenced.

 

Sample output:

 

alfa:rfilenameext column=table:state, timestamp=1604493139455, value=\x08\x00
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:regioninfo, timestamp=1604388258225, value={ENCODED => 35925292c25898671e5a894ce387e167, NAME => 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.', STARTKEY => '', ENDKEY => '0'}
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:seqnumDuringOpen, timestamp=1601269814633, value=\x00\x00\x00\x00\x00\x00\x008
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:server, timestamp=1601269814633, value=ba-wtmp04.asgardalfa.hq.com:16020
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:serverstartcode, timestamp=1601269814633, value=1601061167123
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:sn, timestamp=1604388258050, value=ba-wtmp04.asgardalfa.hq.com,16020,1601061167123
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:state, timestamp=1604388258225, value=CLOSED
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:regioninfo, timestamp=1600969938610, value={ENCODED => 787d1455b84f2d846ce6089392f01fd2, NAME => 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.', STARTKEY => '0', ENDKEY => '1'}
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:seqnumDuringOpen, timestamp=1600780187722, value=\x00\x00\x00\x00\x00\x02^\xBB
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:server, timestamp=1600780187722, value=ba-wtmp08.asgardalfa.hq.com:16020
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:serverstartcode, timestamp=1600780187722, value=1600780162556
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:sn, timestamp=1600969938610, value=ba-wtmp07.asgardalfa.hq.com,16020,1600936054386
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:state, timestamp=1600969938610, value=OPENING
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:regioninfo, timestamp=1601060563980, value={ENCODED => aa9d89b40a9def31a080fdd1776acb4e, NAME => 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.', STARTKEY => '1', ENDKEY => '2'}
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:seqnumDuringOpen, timestamp=1600780186976, value=\x00\x00\x00\x00\x00\x02^\xA9
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:server, timestamp=1600780186976, value=dr1-wtmp02.asgardalfa.hq.com:16020
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:serverstartcode, timestamp=1600780186976, value=1600780163021
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:sn, timestamp=1601060563980, value=ba-wtmp05.asgardalfa.hq.com,16020,1601049145467

 

I have been scanning various sources but these have not been very clear or relevant.  For this problem I just want to remove all references (rows) which are related to alfa:rfilenameext table from the hbase:meta table.  Whichever way this happens is of no importance.

 

However, there are other tables in existence on this cluster which are needed so I am not sure about a rebuild of the entire meta table.

 

Apologies in advance...I am a complete hbase newbie and was hoping there was a command such as:

 

delete 'alfa:rfilenameext' from 'hbase:meta'

 

which might serve to remove all rows for that table.

smdas
smdasAnswer
Gold ⭐️⭐️
December 9, 2020

Hello @TGH 

 

Thanks for the response. To your queries,

 

(I) HBCK2 has extraRegionsInMeta for removing Regions from HBase:Meta, which doesn't have any HDFS Directories. Running the HBCK2 Tool with the concerned command shows the Regions in Meta, which aren't present in HDFS & adding a Fix flag (-f) remove them as well. 

(II) Using Delete Command on HBase:Meta isn't an issue, yet we generally avoid making any changes to the HBase:Meta manually. It's more of a recommendation to avoid any manual oversight causing HBase:Meta corruption.

(III) We can change the Region State via HBCK2 setRegionState Command. Note that the HBCK2 Git Page recommend using the Command as a last resort, considering the risky nature. If Customer is aware of the risk associated with the concerned Command, they can run the Command to set the TableState or RegionState.

 

- Smarak

View in thread

12 replies

smdas
Gold ⭐️⭐️
smdas
Gold ⭐️⭐️
December 8, 2020

Hello @TGH 

 

Thanks for using Cloudera Community. You had Region-In-Transition (RIT) & the HDFS Directory has been removed for the Regions along with the ZNode being removed, yet HBase reports RIT. You wish to fix the RIT issue by removing the Meta Table entries as RIT avoids Balancer run. In HBase v2 (CDH v6.3.x), the MasterProcWALs is critical for any Procedure, which are stuck or blocked. You mentioned a lot of procedures (Disable|Delete) being observed. 

 

The graceful manner for your Team to manage the requirement is to use the HBCK2 Tool. You can build the HBCK2 using the Link [1]. Next, You can use the HBCK2 Tool to bypass the Procedure (PIDs) associated with the Table, for which the Region Directories have been removed. Once any PID is bypassed, the HMaster UI Page (Locks & Procedures ) Section would show the PID as "Bypass". After ensuring the required PIDs are bypassed, Restart the HMaster Service & use the HBCK2 Tool to remove the Region entries in Meta, for which the HDFS Region Directories are removed. Use "bypass" & "extraRegionsInMeta" HBCK2 Command as documented in Link [1].

 

Alternatively, You can Stop HMaster > Remove MasterProcWALs (After confirming no RUNNABLE Procedures excluding the PID associated with the Table for which Region Directory have been removed) > Start HMaster. However, this isn't an ideal approach & you can encounter "Master Is Initialising" issue, for which HBCK2 Tool is required. The "Master Is Initialising" context is captured in Link [1] as well.

 

- Smarak

 

[1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2

Translate
TGH
TGHAuthor
Silver ⭐️⭐️⭐️
December 9, 2020

Hi, @smdas,

 

In my post I neglected to mention that (last week) I shut hbase, removed the masterprocwals, dfs removed the structure, zk'd the znodes, and restarted hbase.  This removed all references to the affected table.  There are (currently) no locks or procedures for this table.

 

The hbase UI reports those regions in transition - I assume only because they are contained in hbase:meta - but there is nothing in storage, nothing in zk, and CM shows no table details as that table is not there.

 

I have been poring over info regarding use of hbck2 and cannot seem to find anything resembling a suitable operation that will actually operate on hbase:meta rows, just general options for fixing meta as a kind of global operation.

 

 

Translate
smdas
Gold ⭐️⭐️
smdas
Gold ⭐️⭐️
December 8, 2020

Hello @TGH 

 

Sharing the Steps for building the HBCK2 Jar using Git reference & additionally, refer the Post via [1] for the details on building HBCK2 Tool as well.

 

Screenshot 2020-12-08 at 7.40.24 PM.png

 

- Smarak

 

[1] https://community.cloudera.com/t5/Support-Questions/How-to-get-hbck2-tool-for-CDH-6-3-2/m-p/295867/highlight/true#M218004

Translate
TGH
TGHAuthor
Silver ⭐️⭐️⭐️
December 9, 2020

Getting it and/or building it is not the problem. Does it have the facility to remove redundant meta information, and without corrupting other meta data for needed tables?

 

Why is 'delete' or 'deleteall' unsuitable, is there no way to specify removal of rows containing that table name?  After all, if it is just a table like every other table then there must exist a normal database method to selectively remove data.

 

Is it possible to eradicate transition errors (in this case) by manually altering the state to OPENING -> OPEN -> CLOSING -> CLOSED... would this 'finalise' transition errors to the point they get removed from meta by regular hbase operation?

Translate
smdas
Gold ⭐️⭐️
smdas
Gold ⭐️⭐️
December 18, 2020

Hello @TGH 

 

Yes, After doing any HBCK2 Changes, Restart the Service as the Components have a Cached Version of the Metadata as well. Let us know how things goes. 

 

- Smarak

Translate
Terms & ConditionsAccessibility statement