Hi everyone, as title i'd like to run ANY generic application (written by users) . So the idea is: 1) User write its own Python application (about anything he wants, without using Spark API) 2) I run this app via Spark-submit 3) A SINGLE DOCKER CONTAINER on Yarn is created, with the resources i set
It actually works, but.... Since the users dont use Spark, the Executors "role" in the Spark mechanism is useless, or better, the user DOESNT NEED THEM, so basically the user app should run only in the Application Master. I tried to set spark.executor.instances to 0 (in spark-submit) but an Error appears during the submit saying that "must be a positive value". So, do you know i there is a way to disable/enable executors and just use AM as i please? (Maybe assign all resources to AM and 0 to executors *?*) Because there will be also user that may want to use Spark, so they will have also executors.