HBase: How to remove corrupted hbase:meta rowkeys containing single commas

[#1]

Hi all,

I've run into an interesting issue where corrupted entries with rowkeys containing a single comma (,) were inserted into hbase:meta. These rows show up in scans, but get, delete, and deleteall operations don't work on them. Example of such a row:

ROW COLUMN+CELL
rowkey, column=info:regioninfo, timestamp=..., value=...

Attempts to remove them with the Java client and HBCK2 (fixMeta) didn't work.

Has anyone dealt with this before? Is there a safe way to remove or repair such dangling/corrupted rowkeys from hbase:meta?

In addition, I have opened this upstream issue reporting the problem: https://issues.apache.org/jira/browse/HBASE-29554

Thanks in advance for any guidance.

[#2]

@EmilKle Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our HBase experts @smdas @blizano @pajoshi  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

@DianaTorres thank you for your prompt reply.
I haven't found a solution to this problem as of now. If you have any suggestions on how to solve this issue and get rid of the corrupted meta entries in the existing cluster, I would highly appreciate it.

Thank you in advance.

[#4]

Hi @shubham_sharma @ale241290xdf @aquilodran @David666 Do you have any insights here? Thanks!

[#5]

Just to add even more input on this, I can see a side effect of it shown in the HBCK report under Inconsistent Regions 

Just to add more input on this, in the HBase master logs I can also see:

<2025-09-16T10:11:27.734+0200> <WARN> <janitor.ReportMakingVisitor>: <INCONSISTENCY: Row name is not equal to serialized info:regioninfo content; row=XXX:XXX,e14ef354e1644a6a2b229ffde05f38d3 XXX:XXX,test,12345.07a983faf71bc641bfbb4e8736d378ac.; See if RegionInfo is referenced in another hbase:meta row? Delete?>

In meta:

XXX:XXX,e14ef354e1644a6a2b229ffde05f38d3 column=info:regioninfo, timestamp=2025-08-26T12:12:21.389, value={ENCODED => 07a983faf71bc641bfbb4e8736d378ac, NAME => 'XXX:XXX,test,12345.07a983faf71bc641bfbb4e8736d378ac.', STARTKEY => 'test', ENDKEY => ''}   

[#6]

Hello,
I prepared two patched that allowed me to resolve this issue by deploying it on an RS that hosts the hbase:meta table. You can find more details in the following two upstream issues I have opened with additional patches used for resolving the issue:
https://issues.apache.org/jira/browse/HBASE-29633
https://issues.apache.org/jira/browse/HBASE-29554

From my investigation this problem was not solvable without the extra patches. I have tested and fixed it in HBase v2.5.10 succesfully.

[#7 · Accepted solution]

@EmilKle FYI

➤This issue typically arises because the comma (,) is a reserved delimiter in HBase for the hbase:meta table structure, used to separate the Table Name, Start Key, and Region ID. When a rowkey is inserted with an unexpected comma, the HBase shell and client API often misinterpret the entry as a malformed region name, causing GET or DELETE commands to route incorrectly or fail validation.

➤Here is how you can approach a safe repair, as traditional methods like HBCK2 fixMeta often bypass these "illegal" keys if they don't follow the expected region naming convention.

1. Use the HBase Shell with Hexadecimal Rowkeys

Standard string-based commands in the shell often fail because the shell parses the comma as a delimiter. Instead, find the exact byte representation of the rowkey and delete it using hexadecimal notation.

1. Find the Hex Key: Run a scan to get the exact bytes of the corrupted row.

scan 'hbase:meta', {ROWPREFIXFILTER => 'rowkey,'}

2. Delete using the binary string: If the rowkey is exactly rowkey,, use the binary notation in the shell:

delete 'hbase:meta', "rowkey\x2C", 'info:regioninfo'

(Note: \x2C is the hex code for a comma).