Is it possible to submit multiple jobs in Spark-Submit ?
I have created multiple jobs to be submitted in the spark-submit
I have to submit these created jobs in a cluster deploy mode
Is it possible to submit the jobs to desired cores in the cluster deploy mode?
like job 1 - core 1
job 2 - cores 2 and 3
Is it possible to schedule the jobs to the desired cores?
Note: Cores should be assigned manually by the user
Well in general you can simply run multiple instances to spark-submit in a shell for loop with dynamic no. of cores.
for i in 1 2 3
spark-submit class /jar --executor-memory 2g --executor-cores 3 --master yarn --deploy-mode cluster
Since cores per container are controlled by Yarn configuration, I believe you will need to set the number of executors and the number of cores per executor based on your Yarn configuration to control how many executors and cores get scheduled. So if you set Yarn to allocate 1 core per container and you want two cores for the job then ask for 2 executors with 1 core each from Spark submit. That should give you two containers with 1 executor each. I don't think Yarn will give you an executor with 2 cores if a container can only have 1 core. But if you can have 8 cores per container then you can have 8 executors with 1 core or 4 executors with 2 cores per container. Of course, you can continue to add executors as long as you your Yarn queue has capacity for more containers.
# Run on a YARN cluster
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --executor-memory 2G --num-executors 2 --executor-cores 1 /path/to/examples.jar