Support Questions
Find answers, ask questions, and share your expertise

Disabling Spark on specific nodes

Rising Star

I have a cluster running Spark on YARN (HDP 2.6). Without loosing YARN capabilities, I need to turn off Spark on certain nodes.

How can I achieve this?

1 REPLY 1

Re: Disabling Spark on specific nodes

Guru

@Fernando Lopez Bello , you can use Yarn Node label feature to achieve your goals.

1) add a node label such as "spark nodes" on hosts where you want keep spark application running.

2) Map "spark nodes" node label with a Yarn Queue such as "SparkQueue".

3) Run your Spark applications from "SparkQueue".

This way you can ensure that Spark application will run on specific hosts you want.

Find few useful links for Node labels feature as below.

https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/NodeLabel.html

https://community.hortonworks.com/articles/72450/node-labels-configuration-on-yarn.html