I did change the parameter "yarn.app.mapreduce.am.resource.mb" to 2 GB (2048 MB).
Although the second Spark job is now running fine under "Fair Scheduler" configuration, the tasks under the second Spark job are not getting the required number of resources at all.
[Stage 0:> (0 + 0) / 1]20/01/09 22:58:01 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 20/01/09 22:58:16 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 20/01/09 22:58:31 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Here are the important information about the cluster:
1. Number of nodes in the Cluster: 2
2. Total amount of memory of Cluster: 15.28 GB
(yarn.nodemanager.resource.memory-mb = 7821 MB
yarn.app.mapreduce.am.resource.mb = 2048 MB
yarn.scheduler.minimum-allocation-mb = 1024 MB
yarn.scheduler.maximum-allocation-mb = 3072 MB)
3. Number of executors set through the program: 5 (spark.num.executors)
4. Number of cores set through the program: 3 (spark.executor.cores)
5. Spark Driver Memory and Spark Executor Memory: 2g each
Please help me in understanding what else is going wrong.
Note: With the same set of parameters (along with yarn.app.mapreduce.am.resource.mb of 1024 MB), the Spark job run fine when YARN Capacity Scheduler is set. However, it doesn't run when YARN Fair Scheduler is set. So, I want to understand what's going wrong only with Fair Scheduler.
From my perspictive you are limiting your default queue to use at minimum 1024MB 0vCores and at maximum 8196MB 0vCores. In both cases no cores are set - when you try to run a job it requires to run with 1024MB memory and 1vCores - it then fails to allocate the 1vCore due to 0vCore min/max restriction and it sends 'exceeds maximum AM resources allowed'
That's why I think the issue is with the core utilization and not with memory.