11-28-2016 01:30 AM
I am using CDH 5.7 for both dev and prod clusters. I've been reading about pre-emption feature in Capacity scheduler and planning to use it for both clusters. Based on my reading, come to know that it is matured feature (released almost 3 years back etc), but haven't found any related docs/articles from cloudera side. I would like to know Cloudera's perspective on enabling this feature in capacity scheduler.
11-29-2016 08:41 PM
We have been using EMR and moved to CDH around 5 months back. Moving from CS to FS requires sometime for us though it is on "todo" list. Also, I do understand your recommendation as Cloudera is making significant progress on FS. I've been doing lot of reading about this as well. Given this situtation and CDH 5.7 has Hadoop 2.6.0+CDH 5.7.0, Can we use pre-emption in capacity scheduler for production cluster given that CS has implemented in version much earlier than Hadoop 2.6.0?
In addition, I am seeing recent fixes in YARN-4752 around 2-3 weeks back. Assuming it has been corporated in CDH later than 5.7 version, it requires upgrade on current running clusters as well? If yes, then it is quite big effort for us.
07-13-2017 10:22 AM
We are looking at https://issues.apache.org/jira/browse/YARN-4752 as well. Can you help confirm which version of CDH picked up YARN-4752? Somehow I couldn't find YARN-4752 mentioned thru release notes at https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_fixed.html#xd_583c10bf....
07-13-2017 10:44 AM
YARN-4752 is in CDH5.11.0. You can find it in realse notes, check this link https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_fixed_in_511.html#fixe...