For adding custom classes to the classpath you should use one of the two following options: - add them via the command line options - add them via the config   For the driver you have the option to use: --driver-class-path /path/to/file   Or for the the executor use --conf "spark.executor.extraClassPath=/path/to/jar" In spark-defaults.conf set the two values (or one if you only need it for one side   spark.driver.extraClassPath   spark.executor.extraClassPath This can be done through the CM UI.   Depending on the exact thing you are doing you might see limitations of which option you can use.   Wilfred ... View more

The only way to use Spark when you do not have a Spark action is to use the shell based action and create the proper spark-submit command for it. You will need to make sure that the configuration and classpath etc is set from the action.   Wilfred ... View more

No sorry I can not deduce that from the data given. Check the RM logs that is whee all scheduling activities are logged.   Wilfred ... View more

If you run spark on yarn you have two modes: client and cluster. If you run in cluster mode the AM also runs the driver. If you run client mode the driver runs locally and there is just an AM to manage the executors. Check running on yarn for the full details. The amount of resources that can be used for AM's is limited, check for the "Max Application Master Share" for the queues.   On top of the memory requirement that you have set for the executor are the heap sizes on top of that the overhead will be added (7% with a minimum of 384MB). That will increase you container request and could see it being round up to 2GB based on the increment value you have set. Check yarn.scheduler.increment-allocation-mb   Like you did you need to check what is available on each node to see if you have room, having space in the cluster does not always mean that you can satisfy the request.   Wilfred ... View more