Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.
Hi all, I have a strange issue related to spark ressources allocation in Yarn. The situation is the following: We have a Cloudera cluster with CDH 5.4.2 and Spark 1.3.0. Regarding this issue there are two main users of the cluster. Thereby the following scheduling issues using Yarn occurs: * When user A runs a couple of spark job, user B has no problems to start additional spark jobs and allocate corresponding ressources * But when user B has at least one spark job running, user A is not able to start any other job, although only half of the cluster ressources are used * Both are running under the default pool, just different users So, my questions would be: * Does anybody has an idea what configuration or other parameters could be the cause? * Are there yarn specific log files regarding the scheduling behavior giving a clue to what happens? Would be great if somebody an idea or a simliar problem. Cheers, Matthias
... View more