Skip to main content
SantoshB
Bronze ⭐️⭐️⭐️
Nov 23, 2022 10:43 PM UTC
Question

WARN cluster.YarnScheduler: Initial job has not accepted any resources

I'm using Cloudera manager 7.4.4 and running the spark(2.4.4) application. I'm facing below warning and application is going on to infinite loop.

 

WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

 

SantoshB_0-1669243075939.png

 

I'm using a simple spark-submit command. spark-submit <filename.py>
my python version is 2.7

When I'm running the below command I got many other errors

spark-submit --master yarn --deploy-mode cluster --driver-memory 5g --executor-memory 5g --num-executors 3 --executor-cores 2 <filename.py>

 

Below are the errors

 

SantoshB_1-1669243319409.png

 

 

2 replies

paras
Gold ⭐️⭐️⭐️
Gold ⭐️⭐️⭐️
Dec 29, 2022 12:03 PM UTC

you should check the container logs for the one mentioned in the snapshots above for more details. Generally caused due to resource crunch. You can consider increasing the below yarn configs as well.

yarn.nodemanager.resource.memory-mb
yarn.nodemanager.resource.cpu-vcores

 

Hope this helps,
Paras
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Translate
AsimShaikh
Gold ⭐️
Gold ⭐️
Jan 2, 2023 2:02 PM UTC

Hi @SantoshB 

You can see such messages when you have reached user factor limit/resource limit on queue level.

 

You can check tune user-limit factor or check queue utilization to schedule applications accordingly. 

 

Also seems application has failed with exit-code 13, can you please share YARN trace to identify reason for failure?

 

 

Translate