Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to choose the queue in which you want to submit the job when using Zeppelin?

Solved Go to solution

How to choose the queue in which you want to submit the job when using Zeppelin?

New Contributor

Hi,

I am using Apache Zeppelin on HDP 2.4 and was wondering how would i choose/change the queue in which my jobs are being submitted? Currently it goes into the default queue but I would like to dedicate a queue for zeppelin alone, and run all jobs submitted from zeppelin there. But I am unable to switch the queue. Can anyone suggest me how to do it?

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to choose the queue in which you want to submit the job when using Zeppelin?

Guru

If you are using zeppelin for spark, you can change JAVA_OPTS in zeppelin-env on zepplien configs and add something like

-Dspark.yarn.queue=my_zeppelin_queuename

You can add mapreduce and tez queues as well in JAVA_OPTS

4 REPLIES 4

Re: How to choose the queue in which you want to submit the job when using Zeppelin?

Guru

If you are using zeppelin for spark, you can change JAVA_OPTS in zeppelin-env on zepplien configs and add something like

-Dspark.yarn.queue=my_zeppelin_queuename

You can add mapreduce and tez queues as well in JAVA_OPTS

Re: How to choose the queue in which you want to submit the job when using Zeppelin?

@vagrawalSetup YARN queue:
  • (Optional) You can setup/configure a YARN queue to customize what portion of the cluster the Spark job should use. To do this follow the two steps below:

    i. Open the Yarn Queue Manager view to setup a queue for Spark with below capacities:

    • Capacity: 50%
    • Max Capacity: 90% (on sandbox, do not reduce below this or the Spark jobs will not run)

screen-shot-2016-05-25-at-92342-pm.png

Re: How to choose the queue in which you want to submit the job when using Zeppelin?

New Contributor

In such case, if we like to define livy.spark.yarn.queue, should it be same as spark.yarn.queue or it could be different queue?

Thank you!

Leonid

,

In such case, can we define livy.spark.yarn.queue to pint to different queue, or it have to be same as spark.yarn.queue?

Re: How to choose the queue in which you want to submit the job when using Zeppelin?

I was able to set Livy queue by just setting livy.spark.yarn.queue=mylivyqueue in the Livy interpreter in Zeppelin and after restarting the Interpreter, Livy notebooks start runnin on that queue. By the way, my spark.yarn.queue=default.

Don't have an account?
Coming from Hortonworks? Activate your account here