Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDFS client fails with "ArrayIndexOutofBoundsException"

avatar
Contributor

We have completely same problem as https://issues.apache.org/jira/browse/HDFS-11797 with HDP 2.6.1.

https://issues.apache.org/jira/browse/HDFS-11797?focusedCommentId=16039577&page=com.atlassian.jira.p...

org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number of corrupt replicas for blk_123456789_123456 blockMap has 0 but corrupt replicas map has 1
 org.apache.hadoop.ipc.Server: IPC Server handler 34 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from xxx.xxx.xxx.xxx:xxxxx Call#91 Retry#0 java.lang.ArrayIndexOutOfBoundsException

Actually, our hive client fails for this problem. and hdfs fsck command also fails for this hdfs file with this problem.

I read a series of JIRA tickets.

https://issues.apache.org/jira/browse/HDFS-9958

https://issues.apache.org/jira/browse/HDFS-10788

https://issues.apache.org/jira/browse/HDFS-11797

https://issues.apache.org/jira/browse/HDFS-11445

https://issues.apache.org/jira/browse/HDFS-11755

At the second last comment of HDFS-11755

https://issues.apache.org/jira/browse/HDFS-11755?focusedCommentId=16200946&page=com.atlassian.jira.p...

As discussed in HDFS-11445, a regression caused by HDFS-11445 is fixed by HDFS-11755. I'd like to backport HDFS-11755 into branch-2.7 as a result.

and,

https://issues.apache.org/jira/browse/HDFS-11755?focusedCommentId=16201164&page=com.atlassian.jira.p...

Filed HDFS-12641 to initiate the discussion.

and https://issues.apache.org/jira/browse/HDFS-12641 is not resolved.

I'm not sure, but HDFS-12641 may be only for CDH???

I've also checked, HDFS-11445 is not included in HDP 2.6.1. but, it's included in HDP 2.6.2.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_release-notes/content/patch_hadoop.html

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_release-notes/content/patch_hadoop.html

So, can someone confirm that our current problem with message "blockMap has 0 but corrupt replicas map has 1" is safely fixed with HDP 2.6.2 with HDFS-11445?

Actually, we plan to upgrade from HDP 2.6.1 to HDP 2.6.2. but I'm worry about that upgrading to HDP 2.6.2 would make new problem such as HDFS-11755 says "a regression caused by HDFS-11445 is fixed by HDFS-11755."...

I've confirmed HDFS-11755 is not included in HDP 2.6.1, HDP 2.6.2.

1 ACCEPTED SOLUTION

avatar
Contributor

After upgrading our cluster from HDP 2.6.1 to HDP 2.6.2, the following WARN message is not output in NameNode's log.

WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number of corrupt replicas for blk_xxxx_xxxx blockMap has 0 but corrupt replicas map has 1

So, the problem seems to be fixed by this upgrading in our clusters.

View solution in original post

1 REPLY 1

avatar
Contributor

After upgrading our cluster from HDP 2.6.1 to HDP 2.6.2, the following WARN message is not output in NameNode's log.

WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number of corrupt replicas for blk_xxxx_xxxx blockMap has 0 but corrupt replicas map has 1

So, the problem seems to be fixed by this upgrading in our clusters.