HDFS client fails with "ArrayIndexOutofBoundsException"

[#1]

We have completely same problem as https://issues.apache.org/jira/browse/HDFS-11797 with HDP 2.6.1.

https://issues.apache.org/jira/browse/HDFS-11797?focusedCommentId=16039577&page=com.atlassian.jira.p...

org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number of corrupt replicas for blk_123456789_123456 blockMap has 0 but corrupt replicas map has 1
 org.apache.hadoop.ipc.Server: IPC Server handler 34 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from xxx.xxx.xxx.xxx:xxxxx Call#91 Retry#0 java.lang.ArrayIndexOutOfBoundsException

Actually, our hive client fails for this problem. and hdfs fsck command also fails for this hdfs file with this problem.

I read a series of JIRA tickets.

https://issues.apache.org/jira/browse/HDFS-9958

https://issues.apache.org/jira/browse/HDFS-10788

https://issues.apache.org/jira/browse/HDFS-11797

https://issues.apache.org/jira/browse/HDFS-11445

https://issues.apache.org/jira/browse/HDFS-11755

At the second last comment of HDFS-11755

https://issues.apache.org/jira/browse/HDFS-11755?focusedCommentId=16200946&page=com.atlassian.jira.p...

As discussed in HDFS-11445, a regression caused by HDFS-11445 is fixed by HDFS-11755. I'd like to backport HDFS-11755 into branch-2.7 as a result.

and,

https://issues.apache.org/jira/browse/HDFS-11755?focusedCommentId=16201164&page=com.atlassian.jira.p...

Filed HDFS-12641 to initiate the discussion.

and https://issues.apache.org/jira/browse/HDFS-12641 is not resolved.

I'm not sure, but HDFS-12641 may be only for CDH???

I've also checked, HDFS-11445 is not included in HDP 2.6.1. but, it's included in HDP 2.6.2.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_release-notes/content/patch_hadoop.html

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_release-notes/content/patch_hadoop.html

So, can someone confirm that our current problem with message "blockMap has 0 but corrupt replicas map has 1" is safely fixed with HDP 2.6.2 with HDFS-11445?

Actually, we plan to upgrade from HDP 2.6.1 to HDP 2.6.2. but I'm worry about that upgrading to HDP 2.6.2 would make new problem such as HDFS-11755 says "a regression caused by HDFS-11445 is fixed by HDFS-11755."...

I've confirmed HDFS-11755 is not included in HDP 2.6.1, HDP 2.6.2.

[#2 · Accepted solution]

After upgrading our cluster from HDP 2.6.1 to HDP 2.6.2, the following WARN message is not output in NameNode's log.

WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number of corrupt replicas for blk_xxxx_xxxx blockMap has 0 but corrupt replicas map has 1

So, the problem seems to be fixed by this upgrading in our clusters.