Created 09-09-2017 01:31 PM
How can i target the mapreduce and spark jobs to a set of nodes associated with nodeLabels.
The blog says specify -Dmapreduce.job.node-label-expression=<node label> for MapReduce Jobs.
On spark, If an application is running on YARN client mode, it is documented as to use spark-submit --conf spark.yarn.executor.nodeLabelExpression=<node label>
Both of them do not work on latest HWX 2.5. Appreciate any help on this.
Thanks!
Created 09-09-2017 10:37 PM
Hi @jpj
https://issues.apache.org/jira/browse/MAPREDUCE-6304 from this we can know that the feature landed in HDP 2.6 version.
Please upgrade your version to HDP 2.6
More instructions on upgrade - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_command-line-upgrade/content/ch_upgrade_...
Created 09-11-2017 12:38 AM
Thanks Sathish, but I am already on version 2.6.0.3-8. A different patch version?
Created 09-11-2017 12:39 AM
Apologies for mentioning HDP2.5.
Created 09-11-2017 07:52 AM
@jpj To the best of my knowledge, MAPREDUCE-6304 is only part of HDP-2.6.2.0. Please check this document to upgrade.