@Luis Valdeavellano Those warnings have nothing to do with your issue, but it is good to fix them anyway. 1. If you don't want your job to use all the resources from the cluster, then define a separate YARN queue for Spark jobs and submit it to that queue. I already assume that you submit Spark job via YARN. Obviously, your job will max out that Spark queue resources, but the resources not assigned to that queue could still be used by others. You still have your problem, but others can still execute jobs. 2. Look at your job and determine why is using so much resources, redesign it, tune it, break it in small pieces, etc. If the job is well tuned then your cluster does not have enough resources. Check resource use during the execution of the job to determine the bottleneck (RAM, CPU, etc).
... View more
