Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Disabling Spark on specific nodes

Disabling Spark on specific nodes

Rising Star

I have a cluster running Spark on YARN (HDP 2.6). Without loosing YARN capabilities, I need to turn off Spark on certain nodes.

How can I achieve this?

1 REPLY 1
Highlighted

Re: Disabling Spark on specific nodes

Guru

@Fernando Lopez Bello , you can use Yarn Node label feature to achieve your goals.

1) add a node label such as "spark nodes" on hosts where you want keep spark application running.

2) Map "spark nodes" node label with a Yarn Queue such as "SparkQueue".

3) Run your Spark applications from "SparkQueue".

This way you can ensure that Spark application will run on specific hosts you want.

Find few useful links for Node labels feature as below.

https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/NodeLabel.html

https://community.hortonworks.com/articles/72450/node-labels-configuration-on-yarn.html