Support Questions
Find answers, ask questions, and share your expertise

Is there a way to control at the code level which YARN queue is selected when running spark commands within Zeppelin?

Explorer

I know it is possible to select a queue through a Livy REST API call, but cannot find an equivalent within Zeppelin itself.

3 REPLIES 3

Guru

@mjohnson I suppose this is what you are looking for

https://community.hortonworks.com/questions/35632/how-to-choose-the-queue-in-which-you-want-to-submi...

basically to add this in JAVA_OPTS within zeppelin-env or zeppelin-config

-Dspark.yarn.queue=my_zeppelin_queuename

Explorer

@Shyam Sunder Rai, I saw that...looking for a parameter like the Livy 'queue' attribute so that I can specify for a specific notebook and paragraph the YARN queue. But I am thinking you are right that the only options is with the spark.yarn.queue instance property.

Guru

@mjohnson, Zeppelin currently supports to set queue at interpreter level. You can set livy.spark.yarn.queue parameter in livy interpreter setting. ( This config will allow to run all Livy notebooks in specific queue )

Currently, Zeppelin does not support different queues for specific notebook / user. ZEPPELIN-1603 is tracking this issue.

; ;