question Re: Spark jobs are stuck under YARN Fair Scheduler in Support Questions

Spark jobs are stuck under YARN Fair Scheduler

ssk26 — Tue, 31 Dec 2019 09:09:09 GMT

Hi,

I have setup YARN Fair-scheduler in Ambari (HDP 3.1.0.0-78) for "Default" queue itself. So far, I haven't added any new queues.

Now, I want to run a simple job against the queue and when I submit the job, the application state is in "ACCEPTED" state forever. I get the below message in YARN logs:

The additional information is given below. Please help me in fixing this issue at the earliest.

For "default" queue, the below parameters are set through "fair-scheduler.xml".

Also, no jobs are currently running, apart from the one that I have launched.

Given below is the screenshot, which confirms that the maximum AM resource percent is greater than 0.1

Re: Spark jobs are stuck under YARN Fair Scheduler

EricL — Thu, 02 Jan 2020 22:01:09 GMT

@ssk26 ,

Can you please check the value of below settings?

yarn.app.mapreduce.am.resource.mb
yarn.app.mapreduce.am.resource.cpu-vcores

Cheers
Eric

Re: Spark jobs are stuck under YARN Fair Scheduler

ssk26 — Fri, 03 Jan 2020 05:20:21 GMT

Hi @EricL,

Thanks for your inputs.

The value of yarn.app.mapreduce.am.resource.mb is set to 1024 in "mapred-site.xml" file.

I was not able to find the value of "yarn.app.mapreduce.am.resource.cpu-vcores" in any of the XML files (i.e core-site.xml, mapred-site.xml, yarn-site.xml, capacity-scheduler.xml etc..)

<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>1024</value>
</property>

Here is the progress that I have made so far:

- After setting the Yarn fair scheduler, I did set the Spark program to use Fair scheduling pool also (from default Spark fair scheduler XML template)

- The minimum and maximum allocation (in MB) for Fair Scheduler in Yarn is set to 1024 MB and 3072 MB respectively.

- After running a single Spark job [with both Driver and Executor memory set to 512MB], I was able to verify that the job is running. But, it was consuming the entire 3 GB memory.

- So, the next Spark job is not running at all, as it is waiting for the memory.

- But, if I revert back the YARN scheduling to "Capacity Scheduler", then with the same memory settings, both the jobs are running fine without any issues.

So, what additional memory related parameters need to be set in Fair Scheduling for the jobs to run properly?

Please help me in fixing this issue.

Re: Spark jobs are stuck under YARN Fair Scheduler

ssk26 — Sun, 05 Jan 2020 17:26:21 GMT

Hi @EricL ,

This is just a gentle reminder.

Can you please help me in fixing this issue?

Thanks and Regards,

Sudhindra

Re: Spark jobs are stuck under YARN Fair Scheduler

ssk26 — Wed, 08 Jan 2020 07:05:21 GMT

Hi @EricL ,

I am still facing the same issue when I use YARN fair scheduler to run the Spark jobs.

With the same memory configuration, the Spark jobs are running fine when YARN Capacity Scheduler is used.

Can you please help me in fixing this issue?

Thanks and Regards,

Sudhindra

Re: Spark jobs are stuck under YARN Fair Scheduler

lyubomirangelo — Wed, 08 Jan 2020 14:43:57 GMT

Hello,

In your screenshot the

Is set to 8192 mb, 0vcore

And your job requires at least 1vcore as seen in the Diagnostics section.

Please try increasing the vcore size in <queueMaxResourcesDefault> and try to run the job again.

Best,
Lyubomir

Re: Spark jobs are stuck under YARN Fair Scheduler

EricL — Thu, 09 Jan 2020 11:00:38 GMT

@ssk26,

Have you tried to increase yarn.app.mapreduce.am.resource.mb? Default value is 1GB, and I see from your screenshot that requested 1GB exceeds AM's limit.

Cheers
Eric

Re: Spark jobs are stuck under YARN Fair Scheduler

ssk26 — Fri, 10 Jan 2020 04:44:58 GMT

Hi @EricL ,

I did change the parameter "yarn.app.mapreduce.am.resource.mb" to 2 GB (2048 MB).

Although the second Spark job is now running fine under "Fair Scheduler" configuration, the tasks under the second Spark job are not getting the required number of resources at all.

[Stage 0:> (0 + 0) / 1]20/01/09 22:58:01 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
20/01/09 22:58:16 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
20/01/09 22:58:31 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Here are the important information about the cluster:

1. Number of nodes in the Cluster: 2

2. Total amount of memory of Cluster: 15.28 GB

(yarn.nodemanager.resource.memory-mb = 7821 MB

yarn.app.mapreduce.am.resource.mb = 2048 MB

yarn.scheduler.minimum-allocation-mb = 1024 MB

yarn.scheduler.maximum-allocation-mb = 3072 MB)

3. Number of executors set through the program: 5 (spark.num.executors)

4. Number of cores set through the program: 3 (spark.executor.cores)

5. Spark Driver Memory and Spark Executor Memory: 2g each

Please help me in understanding what else is going wrong.

Note: With the same set of parameters (along with yarn.app.mapreduce.am.resource.mb of 1024 MB), the Spark job run fine when YARN Capacity Scheduler is set. However, it doesn't run when YARN Fair Scheduler is set. So, I want to understand what's going wrong only with Fair Scheduler.

Re: Spark jobs are stuck under YARN Fair Scheduler

EricL — Fri, 10 Jan 2020 04:56:04 GMT

Have you tried @lyubomirangelo 's suggestion? He noticed that you have 0vcore configured, not sure if that has any negative impacts.

And you only have 2 nodes in the cluster, meaning only one NodeManager? One master one worker node?

Re: Spark jobs are stuck under YARN Fair Scheduler

lyubomirangelo — Fri, 10 Jan 2020 11:54:13 GMT

Hi @ssk26

From my perspictive you are limiting your default queue to use at minimum 1024MB 0vCores and at maximum 8196MB 0vCores. In both cases no cores are set - when you try to run a job it requires to run with 1024MB memory and 1vCores - it then fails to allocate the 1vCore due to 0vCore min/max restriction and it sends 'exceeds maximum AM resources allowed'

That's why I think the issue is with the core utilization and not with memory.

HTH

Best,
Lyubomir

Re: Spark jobs are stuck under YARN Fair Scheduler

ssk26 — Fri, 17 Jan 2020 06:26:58 GMT

Hi @lyubomirangelo and @EricL ,

Sorry for the delayed response. Thanks for your inputs.

I have already changed the number of vcores. But, I am still facing the same issue.

In the meantime, I was able to execute the jobs with YARN Capacity scheduler (with the same memory configuration). So, I am not sure what's wrong with the settings of YARN Fair Scheduler.

Please suggest if any specific settings are required for YARN Fair Scheduler.

Also, I am still using default queue. I haven't set a separate Queue for handling fair scheduler.

Thanks and Regards,

Sudhindra

Re: Spark jobs are stuck under YARN Fair Scheduler

lyubomirangelo — Fri, 17 Jan 2020 10:05:13 GMT

Hi Sudhinra,

Thank you for the update.

Can you share the SparkConf you use for your applications;

The following settings should work for small resource apps (Note dynamic allocation is disabled):

conf = (SparkConf().setAppName("simple")
        .set("spark.shuffle.service.enabled", "false")
        .set("spark.dynamicAllocation.enabled", "false")
        .set("spark.cores.max", "1")
        .set("spark.executor.instances","2")
        .set("spark.executor.memory","200m")
        .set("spark.executor.cores","1")

From:

https://stackoverflow.com/questions/44581585/warn-cluster-yarnscheduler-initial-job-has-not-accepted-any-resources

PS: Share the number of cores available on your nodes, spark.executor.cores should not be higher than number of cores available on each node. Also, are you running spark in cluster or client mode?

HTH

Best,
Lyubomir

Re: Spark jobs are stuck under YARN Fair Scheduler

lyubomirangelo — Fri, 17 Jan 2020 12:36:17 GMT

Hi Sudhnidra,

Please take a look at:
https://blog.cloudera.com/yarn-fairscheduler-preemption-deep-dive/
https://blog.cloudera.com/untangling-apache-hadoop-yarn-part-3-scheduler-concepts/
https://clouderatemp.wpengine.com/blog/2016/06/untangling-apache-hadoop-yarn-part-4-fair-scheduler-queue-basics/

What type of FairScheduler are you using:

Steady FairShare
Instantaneous FairShare

What is the weight of the default queue you are submitting your apps to?

Best,
Lyubomir