https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134913#M35048 <P>Thanks Rajkumar for quick reply. </P><P>I have increased value for spark.executor.memory but not sure how to change value for spark.memory.storageFraction. Please suggest the location to find spark.memory.storageFraction.</P><P>Thanks,</P><P>Yoges</P> Tue, 19 Jul 2016 14:12:50 GMT yrajak 2016-07-19T14:12:50Z https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134911#M35046 <P>Hi,</P><P>I have downloaded HDP 2.4 sandbox for evaluation of new memory management. Below are the environment details </P><P>- Windows Desktop (64 bit) with 16 GB memory</P><P>- Oracle VM player</P><P>- HDP 2.4 (Sandbox) </P><P>I started the VM and checked the executors for Spark Thrift server (port 4040). The value for Storage Memory is 511.5 MB. That means total Java heap size is ~1GB. </P><P>Kindly let me know how can I increase the Java Heap size and setting for other dependent service/configuration. </P><P>Thanks,</P><P>Yogesh</P> Tue, 19 Jul 2016 14:04:34 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134911#M35046 yrajak 2016-07-19T14:04:34Z https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134912#M35047 <P>increase spark.executor.memory to the desired and then adjust spark.memory.storageFraction value which is default .5 (with this default value .9*.5*executor memory will be used as storage memory)</P> Tue, 19 Jul 2016 14:08:16 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134912#M35047 rajkumar_singh 2016-07-19T14:08:16Z https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134913#M35048 <P>Thanks Rajkumar for quick reply. </P><P>I have increased value for spark.executor.memory but not sure how to change value for spark.memory.storageFraction. Please suggest the location to find spark.memory.storageFraction.</P><P>Thanks,</P><P>Yoges</P> Tue, 19 Jul 2016 14:12:50 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134913#M35048 yrajak 2016-07-19T14:12:50Z https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134914#M35049 <P> Example from the Spark doc page (http://spark.apache.org/docs/latest/submitting-applications.html)</P> <PRE># Run on a Spark standalone cluster in cluster deploy mode with supervise ./bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://207.184.161.138:7077 \ --deploy-mode cluster \ --supervise \ --executor-memory 20G \ --total-executor-cores 100 \ /path/to/examples.jar \ 1000</PRE><P>executor-memory is what you want to adapt</P> Tue, 19 Jul 2016 14:14:59 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134914#M35049 bwalter1 2016-07-19T14:14:59Z https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134915#M35050 <P>you can update this in conf/spark-env.sh file and also overwrite it with your job options like this</P><P>--conf spark.storage.memoryFraction=0.4</P> Tue, 19 Jul 2016 14:30:05 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/HDP-2-4-Sandbox-How-to-check-JVM-size-and-how-to-allocate/m-p/134915#M35050 rajkumar_singh 2016-07-19T14:30:05Z