Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Yarn ressource allocation issue

Highlighted

Spark Yarn ressource allocation issue

New Contributor

Hi all,

 

I have a strange issue related to spark ressources allocation in Yarn. The situation is the following: We have a Cloudera cluster with CDH 5.4.2 and Spark 1.3.0. Regarding this issue there are two main users of the cluster. Thereby the following scheduling issues using Yarn occurs:

 

* When user A runs a couple of spark job, user B has no problems to start additional spark jobs and allocate corresponding ressources

* But when user B has at least one spark job running, user A is not able to start any other job, although only half of the cluster ressources are used

* Both are running under the default pool, just different users

 

So, my questions would be:

* Does anybody has an idea what configuration or other parameters could be the cause?

* Are there yarn specific log files regarding the scheduling behavior giving a clue to what happens?

 

Would be great if somebody an idea or a simliar problem.

 

Cheers,

Matthias