HDFS client fails with "ArrayIndexOutofBoundsException"

tomo_hirano — Sat, 21 Oct 2017 00:23:26 GMT

We have completely same problem as https://issues.apache.org/jira/browse/HDFS-11797 with HDP 2.6.1.

https://issues.apache.org/jira/browse/HDFS-11797?focusedCommentId=16039577&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16039577

org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number of corrupt replicas for blk_123456789_123456 blockMap has 0 but corrupt replicas map has 1
 org.apache.hadoop.ipc.Server: IPC Server handler 34 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from xxx.xxx.xxx.xxx:xxxxx Call#91 Retry#0 java.lang.ArrayIndexOutOfBoundsException

Actually, our hive client fails for this problem. and hdfs fsck command also fails for this hdfs file with this problem.

I read a series of JIRA tickets.

https://issues.apache.org/jira/browse/HDFS-9958

https://issues.apache.org/jira/browse/HDFS-10788

https://issues.apache.org/jira/browse/HDFS-11797

https://issues.apache.org/jira/browse/HDFS-11445

https://issues.apache.org/jira/browse/HDFS-11755

At the second last comment of HDFS-11755

https://issues.apache.org/jira/browse/HDFS-11755?focusedCommentId=16200946&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16200946

As discussed in HDFS-11445, a regression caused by HDFS-11445 is fixed by HDFS-11755. I'd like to backport HDFS-11755 into branch-2.7 as a result.

and,

https://issues.apache.org/jira/browse/HDFS-11755?focusedCommentId=16201164&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16201164

Filed HDFS-12641 to initiate the discussion.

and https://issues.apache.org/jira/browse/HDFS-12641 is not resolved.

I'm not sure, but HDFS-12641 may be only for CDH???

I've also checked, HDFS-11445 is not included in HDP 2.6.1. but, it's included in HDP 2.6.2.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_release-notes/content/patch_hadoop.html

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_release-notes/content/patch_hadoop.html

So, can someone confirm that our current problem with message "blockMap has 0 but corrupt replicas map has 1" is safely fixed with HDP 2.6.2 with HDFS-11445?

Actually, we plan to upgrade from HDP 2.6.1 to HDP 2.6.2. but I'm worry about that upgrading to HDP 2.6.2 would make new problem such as HDFS-11755 says "a regression caused by HDFS-11445 is fixed by HDFS-11755."...

I've confirmed HDFS-11755 is not included in HDP 2.6.1, HDP 2.6.2.

Re: HDFS client fails with "ArrayIndexOutofBoundsException"

tomo_hirano — Mon, 13 Nov 2017 16:04:17 GMT

After upgrading our cluster from HDP 2.6.1 to HDP 2.6.2, the following WARN message is not output in NameNode's log.

WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number of corrupt replicas for blk_xxxx_xxxx blockMap has 0 but corrupt replicas map has 1

So, the problem seems to be fixed by this upgrading in our clusters.

question HDFS client fails with "ArrayIndexOutofBoundsException" in Archives of Support Questions (Read Only)

HDFS client fails with "ArrayIndexOutofBoundsException"

Re: HDFS client fails with "ArrayIndexOutofBoundsException"