Getting error in region server

[#1]

2017-04-29 14:24:14,135 ERROR [B.fifo.QRpcServer.handler=49,queue=1,port=16020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalStateException: Invalid currTagsLen -32712. Block offset: 3707853, block length: 72841, position: 0 (without header). at org.apache.hadoop.hbase.io.hfile.HFileReaderV3$ScannerV3.checkTagsLen(HFileReaderV3.java:226)

My application is using hbase counter and I am seeing this issue today. Can someone tell me what this means and how to fix it?

I am not not using any tags feature. Let me know what extra info is required.

[#2]

Have you verified that HDFS is healthy? e.g. HDFS FSCK

What operation triggers this error? Does the error happen every time you run that operation?

[#3]

hdfs fsck says it is healthy.

The filesystem under path '/apps/hbase' is HEALTHY

The Increment operation from the application side is triggering this error. And it is happening every time. It feels that one particular HFile block is corrupt (with negative tagLength).

hbase(main):007:0> scan 'counters', {LIMIT=>1,STARTROW=>'ad:event_count:a'}
Returning the result

hbase(main):008:0> scan 'counters', {LIMIT=>1,STARTROW=>'ad:event_count:b'}
ROW                                                         COLUMN+CELL                                                                                                    
ERROR: java.io.IOException: java.lang.IllegalStateException: Invalid currTagsLen -32701. Block offset: 272031, block length: 72441, position: 32487 (without header).
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleException(HRegion.java:5607)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5579)
	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2627)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2613)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2595)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2282)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295)

[#4]

Hi,Do you resolved this problem?

[#5]

@balance002, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

[#6 · Accepted solution]

Hello @balance002 

The Error "IllegalStateException: Invalid currTagsLen" likely indicates Hfile Corruption & a possible factor is too many cell-level ACL tags being applied to a specific table. Unfortunately, such Corruption requires additional review into the Hfile Metadata & is extremely rarely observed. As such, It's best to open a Support Case with Cloudera, if you are experiencing the concerned issue to avoid sharing any Hfile Metadata (Which includes Customer's Data) over Community. 

As a last resort, We need to use the HFile tool with the -p or -k flags, and determine which HFile the command fails on. The -p flag will print all of the data scanned, while -k will only check the integrity of the row, but both will fail on the affected HFile, indicating which has the affected cell:

(1) Find the bad HFile with: hfile -k <HFile location>
(2) Disable the table: disable '<table name>'
(2) Move the HFile to a different location in HDFS: hdfs dfs -mv <Hfile location> <tmp location>
(3) Reload all of the data from that HFile, applying the tags to those rows so that they are not too large

Regards, Smarak

[#7]

Hello @balance002 

Kindly let us know if the above Post dated 2022-11-08 assisted your team to identify the affected Hfile(s) & take necessary remediation to ensure Table's access is restored for Users.

Regards, Smarak