Hello @loridigia, I have tried to run the sample job like below and I see only one executor and one Driver container. # cd /usr/hdp/current/spark2-client # su spark $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --num-executors 1 --driver-memory 512m --executor-memory 512m --executor-cores 2 examples/jars/spark-examples*.jar 10000   Even the spark-shell is also limiting the containers to one when we are mentioning the --num-executors 1 $ spark-shell --num-executors 1   - What is the spark-submit command you are trying to run? - Are you seeing the same issue with the above sample job? ... View more

Hello @loridigia  I don't think there is a direct way to achieve this. But we have a workaround to do that. We can start the Spark jobs with Dynamic Allocation enabled. And we can set the Minimum executors to "0", initial executors to "1" and the idle timeout to "5s". With these configurations, the Spark job will start with 1 executor and after 5 seconds that container will be killed as it will be idle for more than 5 seconds. Now, we will have a Spark application only with the Driver / ApplicationMaster container running.   CONFIGS: --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.executorIdleTimeout=5s --conf spark.dynamicAllocation.initialExecutors=1 --conf spark.dynamicAllocation.maxExecutors=1 --conf spark.dynamicAllocation.minExecutors=1   NOTE: We can add these configs to the spark-defaults.conf so that the changes will be applied to all the Running jobs. Please be careful with other / actual Spark job configurations.   Make sure to mark the answer as the accepted solution. If it resolves your issue !   ... View more

@Michael Bronson The warning message means that the Executor is unable to send the Heartbeat to the driver (might be network issue). This is just a warning message, but each failure increments heartbeat failure count and when we hit the maximum failures the executor will fail and exit with error. There are two configurations that we can tune to avoid this issue. spark.executor.heartbeat.maxFailures (default value: 60) Number of times an executor will try to send heartbeats to the driver before it gives up and exits (with exit code 56). spark.executor.heartbeatInterval ( default value: 10s ) Interval between each executor's heartbeats to the driver. Heartbeats let the driver know that the executor is still alive and update it with metrics for in-progress tasks. spark.executor.heartbeatInterval should be significantly less than spark.network.timeout ... View more

@Bhanu Pamu It seems this is because of the Bug (https://issues.apache.org/jira/browse/AMBARI-22290). You can disable the HDF repository to workaround this issue simply. ... View more