Support Questions
Find answers, ask questions, and share your expertise

Re: Initial job has not accepted any resources

Super Guru

can you post the spark-submit you listed?

Re: Initial job has not accepted any resources

@Timothy Spann

I tried different command to submit the job

spark-submit --class com.Spark.MainClass /home/Test-0.0.1-SNAPSHOT.jar

and

spark-submit --class com.Spark.MainClass -master yarn-client  /home/Test-0.0.1-SNAPSHOT.jar

Re: Initial job has not accepted any resources

Super Guru

You really need more cores. But 2 may work.

spark-submit --class "com.stuff.Class" \ --master yarn --deploy-mode client --driver-memory 1024m --executor-memory 1024m --conf spark.ui.port=4244 MyJar.jar

remove this from your code

.set("spark.master","yarn-client");

add this

sparkConf.set("spark.cores.max", "1")

sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)

sparkConf.set("spark.sql.tungsten.enabled", "true")

sparkConf.set("spark.eventLog.enabled", "true")

sparkConf.set("spark.app.id", "YourId")

Re: Initial job has not accepted any resources

@Timothy Spann

Can you please explain to me what the following line mean

sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)

Re: Initial job has not accepted any resources

Super Guru

KryoSerializer is pretty awesome. It is a faster Java serializer. This will speed up Spark, not related to your issue, but I like to add that to all my Spark projects. When RDDs are in memory they are serialized objects. So a faster, smaller serialization will help with speed and memory.

Re: Initial job has not accepted any resources

I have faced this issue numerous times as well:

"“WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources”

The problem was with Dynamic Resource Allocation over allocating. After turning off Dynamic Resource allocation and then specifying number of executors, executor memory, and cores, my jobs were running.

Turn off Dynamic Resource Allocation:

conf = (SparkConf()

.setAppName("my_job_name")

.set("spark.shuffle.service.enabled", "false")

.set("spark.dynamicAllocation.enabled", "false")

.set("spark.io.compression.codec", "snappy")

.set("spark.rdd.compress", "true"))

sc = SparkContext(conf = conf)

Give values with spark submit (you could also set these in SparkConf as well):

/usr/hdp/2.3.4.0-3485/spark/bin/spark-submit --master yarn --deploy-mode client /home/ec2-user/scripts/validate_employees.py --driver-memory 3g --executor-memory 3g --num-executors 4 --executor-cores 2