HDFS performance issue with HDP-2.6

[#1]

We have faced a HDFS performance issue with HDP-2.6. When i/o intensive jobs are submitted, many BlockSender threads are waited and the number of ThreadsBlocked raises. Then, the performance is down.

We investigated the cause and found that the bug is generated by HDFS-11160 and it fixed by HDFS-11187 which isn't included in the latest HDP-2.6.

Is there any plan to release a new HDP-2.6? If there is the plan, I hope it includes HDFS-11187.

(BTW, HDP-3.0 won't occur the issue since it includes HDFS-11187.)

[#2]

Is there any plan to release a new HDP-2.6?

-No, I think.

[#3]

Thanks for your reply, @schhabra.

[#4 · Accepted solution]

Hi @Takanobu Asanuma ,

Happened to see in Hadoop-HDFS code repo. the issue you are referring to is Fixed in HDP-3.0.1. so I assume HDP-3.0 even don't have the fix.

also cant find it in fixed issues in hortonworks docs : https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/release-notes/content/fixed_issues.html

i would suggest you go ahead and create a Backport Request in hortonworks support portal which will be analyzed by engineers and provided a hotfix if the backport ing is possible?

seeing the code I think backporting won't be an issue.

Please login and accept this answer if you find this helpful 🙂

[#5]

Thanks for your comments, @Akhil S Naik.

IIUC, HDP-3.0 is HDP-3.0.0.0-1634 which includes the fix.
https://github.com/hortonworks/hadoop-release/commit/2021f4bdce3b27c46edaad198f0007a26a8a1391

We will consider the backport request.

I'd like to accept your answer. Thanks again.