Support Questions
Find answers, ask questions, and share your expertise

Unnecessary daemons for spark-shell

New Contributor

I'm trying use latest sandbox in a Virtualbox on Win7(32bit)+2CPU. It is rather poor machine that the sandbox is very very slow, e.g. load average is constantly >20. I would like to stop all unnecessary daemons just to try spark-shell CLI.

YARN, Hive, Oozie and HDFS are most heavy daemons but I cannot find out if spark-shell need them or not.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Unnecessary daemons for spark-shell

Yes, you can stop Oozie and Hive, also HBase and Ranger if they are running. Then, depending on how much RAM do you have on your laptop, you can check and reduce Base Memory in your VM's Settings-->System. The default for Sandbox on Mac is 8G, not sure about Windows. If you are just after spark-shell you can also stop Ambari (ambari-server and ambari-agent).

View solution in original post

2 REPLIES 2

Re: Unnecessary daemons for spark-shell

Explorer

Basically Yarn and HDFS are required if you want to run spark shell on Yarn, for others it depends on your workload.

Re: Unnecessary daemons for spark-shell

Yes, you can stop Oozie and Hive, also HBase and Ranger if they are running. Then, depending on how much RAM do you have on your laptop, you can check and reduce Base Memory in your VM's Settings-->System. The default for Sandbox on Mac is 8G, not sure about Windows. If you are just after spark-shell you can also stop Ambari (ambari-server and ambari-agent).

View solution in original post