question spark shell - lack of memory in Support Questions

spark shell - lack of memory

oriane_tchepgag — Thu, 23 Feb 2017 19:45:15 GMT

I tried for the first time the command "spark-shell" on git bash in on order to learn how to manage data with spark and hive.

I have faced a problem a memory as you can see on the picture.spark-shell.png

I am on sandbox docker hdp 2.5 trying to learn with the "hello world" case.

I have allocated 5GO of ram for my sandbox and my PC have 8GO of RAM in total.

Until now , I have not faced a memory problem

Is it compulsory to add RAM or is there a simple solution for this problem.

Re: spark shell - lack of memory

sandyy006 — Thu, 23 Feb 2017 21:36:35 GMT

@Oriane

Try stopping the services which are not in use. This helps in freeing up the memory so that you can use spark.

Re: spark shell - lack of memory

Jaraxal — Thu, 23 Feb 2017 22:51:39 GMT

@Oriane

Unfortunately the Sandbox has a minimum requirement of 8GB of RAM. It tends to run better with 10-12GB. Given that your computer only has 8GB of RAM total, you are going to have issues trying to run the Sandbox.

As @Sandeep Nemuri mentioned, you can try to turn off HDP components that you don't need to use. This will save some memory, but I believe you'll still have issues.

Re: spark shell - lack of memory

oriane_tchepgag — Mon, 27 Feb 2017 19:23:14 GMT

Thanks @Sandeep. You are rigth for a quick solution.

Re: spark shell - lack of memory

oriane_tchepgag — Mon, 27 Feb 2017 19:23:59 GMT

Thanks @michael.

You are rigth for a long term solution.