Support Questions
Find answers, ask questions, and share your expertise

spark shell - lack of memory

I tried for the first time the command "spark-shell" on git bash in on order to learn how to manage data with spark and hive.

I have faced a problem a memory as you can see on the picture.spark-shell.png

I am on sandbox docker hdp 2.5 trying to learn with the "hello world" case.

I have allocated 5GO of ram for my sandbox and my PC have 8GO of RAM in total.

Until now , I have not faced a memory problem

Is it compulsory to add RAM or is there a simple solution for this problem.

1 ACCEPTED SOLUTION

Accepted Solutions

Explorer

@Oriane

Unfortunately the Sandbox has a minimum requirement of 8GB of RAM. It tends to run better with 10-12GB. Given that your computer only has 8GB of RAM total, you are going to have issues trying to run the Sandbox.

As @Sandeep Nemuri mentioned, you can try to turn off HDP components that you don't need to use. This will save some memory, but I believe you'll still have issues.

View solution in original post

4 REPLIES 4

@Oriane

Try stopping the services which are not in use. This helps in freeing up the memory so that you can use spark.

Thanks @Sandeep. You are rigth for a quick solution.

Explorer

@Oriane

Unfortunately the Sandbox has a minimum requirement of 8GB of RAM. It tends to run better with 10-12GB. Given that your computer only has 8GB of RAM total, you are going to have issues trying to run the Sandbox.

As @Sandeep Nemuri mentioned, you can try to turn off HDP components that you don't need to use. This will save some memory, but I believe you'll still have issues.

View solution in original post

Thanks @michael.

You are rigth for a long term solution.