question Re: Getting error in region server in Archives of Support Questions (Read Only)

Getting error in region server

abhishek_k_work — Wed, 03 May 2017 12:16:40 GMT

2017-04-29 14:24:14,135 ERROR [B.fifo.QRpcServer.handler=49,queue=1,port=16020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalStateException: Invalid currTagsLen -32712. Block offset: 3707853, block length: 72841, position: 0 (without header). at org.apache.hadoop.hbase.io.hfile.HFileReaderV3$ScannerV3.checkTagsLen(HFileReaderV3.java:226)

My application is using hbase counter and I am seeing this issue today. Can someone tell me what this means and how to fix it?

I am not not using any tags feature. Let me know what extra info is required.

Re: Getting error in region server

elserj — Wed, 03 May 2017 22:11:30 GMT

Have you verified that HDFS is healthy? e.g. HDFS FSCK

What operation triggers this error? Does the error happen every time you run that operation?

Re: Getting error in region server

abhishek_k_work — Thu, 04 May 2017 09:00:36 GMT

hdfs fsck says it is healthy.

The filesystem under path '/apps/hbase' is HEALTHY

The Increment operation from the application side is triggering this error. And it is happening every time. It feels that one particular HFile block is corrupt (with negative tagLength).

hbase(main):007:0> scan 'counters', {LIMIT=>1,STARTROW=>'ad:event_count:a'}
Returning the result

hbase(main):008:0> scan 'counters', {LIMIT=>1,STARTROW=>'ad:event_count:b'}
ROW                                                         COLUMN+CELL                                                                                                    
ERROR: java.io.IOException: java.lang.IllegalStateException: Invalid currTagsLen -32701. Block offset: 272031, block length: 72441, position: 32487 (without header).
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleException(HRegion.java:5607)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5579)
	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2627)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2613)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2595)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2282)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295)

Re: Getting error in region server

balance002 — Wed, 28 Sep 2022 07:17:46 GMT

Hi,Do you resolved this problem?

Re: Getting error in region server

VidyaSargur — Wed, 28 Sep 2022 07:49:07 GMT

@balance002, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

Re: Getting error in region server

smdas — Wed, 09 Nov 2022 07:05:54 GMT

Hello @balance002

The Error "IllegalStateException: Invalid currTagsLen" likely indicates Hfile Corruption & a possible factor is too many cell-level ACL tags being applied to a specific table. Unfortunately, such Corruption requires additional review into the Hfile Metadata & is extremely rarely observed. As such, It's best to open a Support Case with Cloudera, if you are experiencing the concerned issue to avoid sharing any Hfile Metadata (Which includes Customer's Data) over Community.

As a last resort, We need to use the HFile tool with the -p or -k flags, and determine which HFile the command fails on. The -p flag will print all of the data scanned, while -k will only check the integrity of the row, but both will fail on the affected HFile, indicating which has the affected cell:

(1) Find the bad HFile with: hfile -k <HFile location>
(2) Disable the table: disable '<table name>'
(2) Move the HFile to a different location in HDFS: hdfs dfs -mv <Hfile location> <tmp location>
(3) Reload all of the data from that HFile, applying the tags to those rows so that they are not too large

Regards, Smarak

Re: Getting error in region server

smdas — Wed, 16 Nov 2022 05:13:18 GMT

Hello @balance002

Kindly let us know if the above Post dated 2022-11-08 assisted your team to identify the affected Hfile(s) & take necessary remediation to ensure Table's access is restored for Users.

Regards, Smarak