Created 05-03-2017 05:16 AM
2017-04-29 14:24:14,135 ERROR [B.fifo.QRpcServer.handler=49,queue=1,port=16020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalStateException: Invalid currTagsLen -32712. Block offset: 3707853, block length: 72841, position: 0 (without header). at org.apache.hadoop.hbase.io.hfile.HFileReaderV3$ScannerV3.checkTagsLen(HFileReaderV3.java:226)
My application is using hbase counter and I am seeing this issue today. Can someone tell me what this means and how to fix it?
I am not not using any tags feature. Let me know what extra info is required.
Created 11-08-2022 11:05 PM
Hello @balance002
The Error "IllegalStateException: Invalid currTagsLen" likely indicates Hfile Corruption & a possible factor is too many cell-level ACL tags being applied to a specific table. Unfortunately, such Corruption requires additional review into the Hfile Metadata & is extremely rarely observed. As such, It's best to open a Support Case with Cloudera, if you are experiencing the concerned issue to avoid sharing any Hfile Metadata (Which includes Customer's Data) over Community.
As a last resort, We need to use the HFile tool with the -p or -k flags, and determine which HFile the command fails on. The -p flag will print all of the data scanned, while -k will only check the integrity of the row, but both will fail on the affected HFile, indicating which has the affected cell:
(1) Find the bad HFile with: hfile -k <HFile location>
(2) Disable the table: disable '<table name>'
(2) Move the HFile to a different location in HDFS: hdfs dfs -mv <Hfile location> <tmp location>
(3) Reload all of the data from that HFile, applying the tags to those rows so that they are not too large
Regards, Smarak
Created 05-03-2017 03:11 PM
Have you verified that HDFS is healthy? e.g. HDFS FSCK
What operation triggers this error? Does the error happen every time you run that operation?
Created 05-04-2017 02:00 AM
hdfs fsck says it is healthy.
The filesystem under path '/apps/hbase' is HEALTHY
The Increment operation from the application side is triggering this error. And it is happening every time. It feels that one particular HFile block is corrupt (with negative tagLength).
hbase(main):007:0> scan 'counters', {LIMIT=>1,STARTROW=>'ad:event_count:a'} Returning the result
hbase(main):008:0> scan 'counters', {LIMIT=>1,STARTROW=>'ad:event_count:b'} ROW COLUMN+CELL ERROR: java.io.IOException: java.lang.IllegalStateException: Invalid currTagsLen -32701. Block offset: 272031, block length: 72441, position: 32487 (without header). at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.handleException(HRegion.java:5607) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5579) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2627) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2613) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2595) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2282) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32295)
Created 09-28-2022 12:17 AM
Hi,Do you resolved this problem?
Created 09-28-2022 12:49 AM
@balance002, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
Regards,
Vidya Sargur,Created 11-08-2022 11:05 PM
Hello @balance002
The Error "IllegalStateException: Invalid currTagsLen" likely indicates Hfile Corruption & a possible factor is too many cell-level ACL tags being applied to a specific table. Unfortunately, such Corruption requires additional review into the Hfile Metadata & is extremely rarely observed. As such, It's best to open a Support Case with Cloudera, if you are experiencing the concerned issue to avoid sharing any Hfile Metadata (Which includes Customer's Data) over Community.
As a last resort, We need to use the HFile tool with the -p or -k flags, and determine which HFile the command fails on. The -p flag will print all of the data scanned, while -k will only check the integrity of the row, but both will fail on the affected HFile, indicating which has the affected cell:
(1) Find the bad HFile with: hfile -k <HFile location>
(2) Disable the table: disable '<table name>'
(2) Move the HFile to a different location in HDFS: hdfs dfs -mv <Hfile location> <tmp location>
(3) Reload all of the data from that HFile, applying the tags to those rows so that they are not too large
Regards, Smarak
Created 11-15-2022 09:13 PM
Hello @balance002
Kindly let us know if the above Post dated 2022-11-08 assisted your team to identify the affected Hfile(s) & take necessary remediation to ensure Table's access is restored for Users.
Regards, Smarak