Support Questions

Find answers, ask questions, and share your expertise

Getting error in region server

avatar
2017-04-29 14:24:14,135 ERROR [B.fifo.QRpcServer.handler=49,queue=1,port=16020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalStateException: Invalid currTagsLen -32712. Block offset: 3707853, block length: 72841, position: 0 (without header). at org.apache.hadoop.hbase.io.hfile.HFileReaderV3$ScannerV3.checkTagsLen(HFileReaderV3.java:226)

My application is using hbase counter and I am seeing this issue today. Can someone tell me what this means and how to fix it?

I am not not using any tags feature. Let me know what extra info is required.

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Hello @balance002 

 

The Error "IllegalStateException: Invalid currTagsLen" likely indicates Hfile Corruption & a possible factor is too many cell-level ACL tags being applied to a specific table. Unfortunately, such Corruption requires additional review into the Hfile Metadata & is extremely rarely observed. As such, It's best to open a Support Case with Cloudera, if you are experiencing the concerned issue to avoid sharing any Hfile Metadata (Which includes Customer's Data) over Community. 

 

As a last resort, We need to use the HFile tool with the -p or -k flags, and determine which HFile the command fails on. The -p flag will print all of the data scanned, while -k will only check the integrity of the row, but both will fail on the affected HFile, indicating which has the affected cell:


(1) Find the bad HFile with: hfile -k <HFile location>
(2) Disable the table: disable '<table name>'
(2) Move the HFile to a different location in HDFS: hdfs dfs -mv <Hfile location> <tmp location>
(3) Reload all of the data from that HFile, applying the tags to those rows so that they are not too large

 

Regards, Smarak

View solution in original post

6 REPLIES 6

avatar
Super Guru

Have you verified that HDFS is healthy? e.g. HDFS FSCK

What operation triggers this error? Does the error happen every time you run that operation?

avatar

hdfs fsck says it is healthy.

The filesystem under path '/apps/hbase' is HEALTHY

The Increment operation from the application side is triggering this error. And it is happening every time. It feels that one particular HFile block is corrupt (with negative tagLength).

hbase(main):007:0> scan 'counters', {LIMIT=>1,STARTROW=>'ad:event_count:a'}
Returning the result
hbase(main):008:0> scan 'counters', {LIMIT=>1,STARTROW=>'ad:event_count:b'}
ROW                                                         COLUMN+CELL                                                                                                    
ERROR: java.io.IOException: java.lang.IllegalStateException: Invalid currTagsLen -32701. Block offset: 272031, block length: 72441, position: 32487 (without header).
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleException(HRegion.java:5607)
	at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5579)
	at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2627)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2613)
	at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2595)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2282)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295)

avatar
Contributor

Hi,Do you resolved this problem?

avatar
Community Manager

@balance002, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Super Collaborator

Hello @balance002 

 

The Error "IllegalStateException: Invalid currTagsLen" likely indicates Hfile Corruption & a possible factor is too many cell-level ACL tags being applied to a specific table. Unfortunately, such Corruption requires additional review into the Hfile Metadata & is extremely rarely observed. As such, It's best to open a Support Case with Cloudera, if you are experiencing the concerned issue to avoid sharing any Hfile Metadata (Which includes Customer's Data) over Community. 

 

As a last resort, We need to use the HFile tool with the -p or -k flags, and determine which HFile the command fails on. The -p flag will print all of the data scanned, while -k will only check the integrity of the row, but both will fail on the affected HFile, indicating which has the affected cell:


(1) Find the bad HFile with: hfile -k <HFile location>
(2) Disable the table: disable '<table name>'
(2) Move the HFile to a different location in HDFS: hdfs dfs -mv <Hfile location> <tmp location>
(3) Reload all of the data from that HFile, applying the tags to those rows so that they are not too large

 

Regards, Smarak

avatar
Super Collaborator

Hello @balance002 

 

Kindly let us know if the above Post dated 2022-11-08 assisted your team to identify the affected Hfile(s) & take necessary remediation to ensure Table's access is restored for Users.

 

Regards, Smarak