question worker uses more ram than it should in Archives of Support Questions (Read Only)

worker uses more ram than it should

ilia987 — Mon, 18 Jun 2018 19:50:42 GMT

We have the following server to act as workers

2*6 cores (24 threads) 64 gb ram based on ambari 2.6.1.5

our process uses approx 1gb, for example when i submit 100 workers with the settings:

spark-submit ..... --executor-memory 2gb

the total ram used us 302 (100*3), because the ram usage is 3 gb, i cant fully use all the computation power, 3*24 >60 (i set the limit to 60) what did i miss?

both answers helped, each improved the ram usage

Re: worker uses more ram than it should

falbani — Mon, 18 Jun 2018 20:11:12 GMT

@ilia kheifets

The difference may come from yarn.scheduler.minimum-allocation-mb, spark memory overhead and jvm. For more information you may want to read the following article: https://blog.csdn.net/oufuji/article/details/50387104

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: worker uses more ram than it should

anarasimham — Mon, 18 Jun 2018 20:42:13 GMT

You may not be accounting for the driver RAM. Spark creates a driver process to act as a "parent" from which the executor processes spawn as separate YARN jobs. You are specifying the executor memory as 2GB but you did not specify the driver's memory limit.

By default, the driver is allocated 1GB of RAM thus explaining your calculations.

https://spark.apache.org/docs/latest/configuration.html

Re: worker uses more ram than it should

ilia987 — Mon, 18 Jun 2018 20:46:52 GMT

setting yarn.scheduler.minimum-allocation-mb to smaller size improved the allocated memory by 30%

Re: worker uses more ram than it should

ilia987 — Mon, 18 Jun 2018 20:58:46 GMT

I have set it to 512M , it works.

when tried to go lower for example 128 i have got an error:

java.lang.IllegalArgumentException: System memory 119537664 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.

Re: worker uses more ram than it should

anarasimham — Mon, 18 Jun 2018 21:13:09 GMT

It looks like the application you've written uses almost 500 MB of driver memory. It sounds like your goal is to utilize all the CPU that your nodes carry - you'll have to either change the way your application works (to reduce the driver RAM) or reduce the executor memory to use all of the threads that your cluster offers.

Re: worker uses more ram than it should

ilia987 — Mon, 18 Jun 2018 21:25:45 GMT

it uses 300-1200mb, but you are right it cpu heavy. and i am trying to maximize the processing power