I have a cluster running Spark on YARN (HDP 2.6). Without loosing YARN capabilities, I need to turn off Spark on certain nodes.
How can I achieve this?
@Fernando Lopez Bello , you can use Yarn Node label feature to achieve your goals.
1) add a node label such as "spark nodes" on hosts where you want keep spark application running.
2) Map "spark nodes" node label with a Yarn Queue such as "SparkQueue".
3) Run your Spark applications from "SparkQueue".
This way you can ensure that Spark application will run on specific hosts you want.
Find few useful links for Node labels feature as below.