question Re: Unnecessary daemons for spark-shell in Archives of Support Questions (Read Only)

Unnecessary daemons for spark-shell

hal_ueda — Thu, 24 Dec 2015 12:32:25 GMT

I'm trying use latest sandbox in a Virtualbox on Win7(32bit)+2CPU. It is rather poor machine that the sandbox is very very slow, e.g. load average is constantly >20. I would like to stop all unnecessary daemons just to try spark-shell CLI.

YARN, Hive, Oozie and HDFS are most heavy daemons but I cannot find out if spark-shell need them or not.

Re: Unnecessary daemons for spark-shell

sshao — Thu, 24 Dec 2015 17:34:12 GMT

Basically Yarn and HDFS are required if you want to run spark shell on Yarn, for others it depends on your workload.

Re: Unnecessary daemons for spark-shell

pminovic — Sat, 26 Dec 2015 09:12:49 GMT

Yes, you can stop Oozie and Hive, also HBase and Ranger if they are running. Then, depending on how much RAM do you have on your laptop, you can check and reduce Base Memory in your VM's Settings-->System. The default for Sandbox on Mac is 8G, not sure about Windows. If you are just after spark-shell you can also stop Ambari (ambari-server and ambari-agent).