Our Spark jobs are failing when run in YARN. When we do run the job, we can see that YARN will create 3 executors - one driver and two others. The exectors are created among three different cluster nodes. When we view the "Executors" page (there is Stages, Environment, Executors tabs), the executors are always created with random ports (15284, 15923, 15222, etc etc)

We have already set spark.executor.ports inside spark-defaults.conf, along with all other known port variables. I don't understand how we can force those executor client ports to be static. Since they are random, the jobs consistently fail due to inability to connect to the ports.

Any ideas on how to manage this without turning off IPTables?