question HDFS performance issue with HDP-2.6 in Archives of Support Questions (Read Only)

HDFS performance issue with HDP-2.6

tasanuma — Thu, 09 Aug 2018 09:38:49 GMT

We have faced a HDFS performance issue with HDP-2.6. When i/o intensive jobs are submitted, many BlockSender threads are waited and the number of ThreadsBlocked raises. Then, the performance is down.

We investigated the cause and found that the bug is generated by HDFS-11160 and it fixed by HDFS-11187 which isn't included in the latest HDP-2.6.

Is there any plan to release a new HDP-2.6? If there is the plan, I hope it includes HDFS-11187.

(BTW, HDP-3.0 won't occur the issue since it includes HDFS-11187.)

Re: HDFS performance issue with HDP-2.6

schhabra1 — Fri, 10 Aug 2018 12:55:56 GMT

Is there any plan to release a new HDP-2.6?

-No, I think.

Re: HDFS performance issue with HDP-2.6

akhilsnaik — Fri, 10 Aug 2018 13:47:09 GMT

Hi @Takanobu Asanuma ,

Happened to see in Hadoop-HDFS code repo. the issue you are referring to is Fixed in HDP-3.0.1. so I assume HDP-3.0 even don't have the fix.

also cant find it in fixed issues in hortonworks docs : https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/release-notes/content/fixed_issues.html

i would suggest you go ahead and create a Backport Request in hortonworks support portal which will be analyzed by engineers and provided a hotfix if the backport ing is possible?

seeing the code I think backporting won't be an issue.

Please login and accept this answer if you find this helpful 🙂

Re: HDFS performance issue with HDP-2.6

tasanuma — Mon, 13 Aug 2018 12:41:05 GMT

Thanks for your reply, @schhabra.

Re: HDFS performance issue with HDP-2.6

tasanuma — Mon, 13 Aug 2018 12:44:56 GMT

Thanks for your comments, @Akhil S Naik.

IIUC, HDP-3.0 is HDP-3.0.0.0-1634 which includes the fix.
https://github.com/hortonworks/hadoop-release/commit/2021f4bdce3b27c46edaad198f0007a26a8a1391

We will consider the backport request.

I'd like to accept your answer. Thanks again.