https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-increase-amount-of-containers-and-executors-when/m-p/163598#M57400 <P><A rel="user" href="https://community.cloudera.com/users/12921/wbekker.html" nodeid="12921">@Ward Bekker</A>, Firstly, find out the correct configuration in spark to occupy a full cluster. You will need to tune num of executors, executor cores, memory, driver memory etc.</P><P>References: </P><P><A href="https://community.hortonworks.com/questions/56240/spark-num-executors-setting.html">https://community.hortonworks.com/questions/56240/spark-num-executors-setting.html</A></P><P><A href="http://stackoverflow.com/questions/37871194/how-to-tune-spark-executor-number-cores-and-executor-memory">http://stackoverflow.com/questions/37871194/how-to-tune-spark-executor-number-cores-and-executor-memory</A></P><P> After figuring out the correct configs, you can use one of the below approaches to set up zeppelin and livy interpreter.</P><P>1) You can set SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.sh to specify number of executors, executor cores, memory, driver memory etc . ( This config will be applied on all the spark &amp; livy interpreters )</P><PRE>export SPARK_SUBMIT_OPTIONS="--num-executors X --executor-cores Y --executor-memory Z"</PRE><P>2) Set configs in livy interpreter. Open livy interpreter page and add below configs in livy interpreter. </P><PRE>livy.spark.executor.instances X livy.spark.executor.cores Y livy.spark.executor.memory Z</PRE> Mon, 20 Mar 2017 03:03:13 GMT yvora 2017-03-20T03:03:13Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-increase-amount-of-containers-and-executors-when/m-p/163597#M57399 <P>When running spark code in Zeppelin via Livy interpreter, I only see a few containers allocated in yarn. What settings do I need to change to make sure I leverage full cluster capacity? I'm using a cluster created by Hortonworks Datacloud on AWS</P> Sat, 18 Mar 2017 20:32:29 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-increase-amount-of-containers-and-executors-when/m-p/163597#M57399 wbekker 2017-03-18T20:32:29Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-increase-amount-of-containers-and-executors-when/m-p/163598#M57400 <P><A rel="user" href="https://community.cloudera.com/users/12921/wbekker.html" nodeid="12921">@Ward Bekker</A>, Firstly, find out the correct configuration in spark to occupy a full cluster. You will need to tune num of executors, executor cores, memory, driver memory etc.</P><P>References: </P><P><A href="https://community.hortonworks.com/questions/56240/spark-num-executors-setting.html">https://community.hortonworks.com/questions/56240/spark-num-executors-setting.html</A></P><P><A href="http://stackoverflow.com/questions/37871194/how-to-tune-spark-executor-number-cores-and-executor-memory">http://stackoverflow.com/questions/37871194/how-to-tune-spark-executor-number-cores-and-executor-memory</A></P><P> After figuring out the correct configs, you can use one of the below approaches to set up zeppelin and livy interpreter.</P><P>1) You can set SPARK_SUBMIT_OPTIONS in conf/zeppelin-env.sh to specify number of executors, executor cores, memory, driver memory etc . ( This config will be applied on all the spark &amp; livy interpreters )</P><PRE>export SPARK_SUBMIT_OPTIONS="--num-executors X --executor-cores Y --executor-memory Z"</PRE><P>2) Set configs in livy interpreter. Open livy interpreter page and add below configs in livy interpreter. </P><PRE>livy.spark.executor.instances X livy.spark.executor.cores Y livy.spark.executor.memory Z</PRE> Mon, 20 Mar 2017 03:03:13 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-increase-amount-of-containers-and-executors-when/m-p/163598#M57400 yvora 2017-03-20T03:03:13Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-increase-amount-of-containers-and-executors-when/m-p/163599#M57401 <P>Thx <A rel="user" href="https://community.cloudera.com/users/551/yvora.html" nodeid="551">@yvora</A> !</P> Mon, 20 Mar 2017 03:59:54 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-increase-amount-of-containers-and-executors-when/m-p/163599#M57401 wbekker 2017-03-20T03:59:54Z