Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Memory allocation to executors in Spark


Memory allocation to executors in Spark

New Contributor

I'm running a Spark application under yarn cluster.

I have 250GB available in my one-node cluster and try to fully utilize it.

I've set for each executor the following:

spark.executor.memory 28672M (= 28G )
spark.yarn.executor.memoryOverhead 2048  (approx 7%)

I expected to see by monitoring with "top" that each executor is utilizing the allocated memory. However, I found that the resident memory is use is ~10GB and the virtual memory is ~30GB.

The yarn log (/var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-host.log) say the same:

2018-03-21 15:09:49,838 INFO  monitor.ContainersMonitorImpl ( - Memory usage of ProcessTree 279993 for container-id container_e09_1521632768675_0005_01_000007: 11.0 GB of 32 GB physical memory used; 31.5 GB of 67.2 GB virtual memory used

I repeated few times with various spark.executor.memory settings. In all my tests, the resident memory in use was less than 40% of my settings, as in the example above.

How can I utilize the entire 32GB that I've allocated ??


Re: Memory allocation to executors in Spark

@Yuval Smp

What value you have for

$ echo /proc/sys/vm/swappiness

If greater than 0, could try to set swappiness to 0 and test?

Re: Memory allocation to executors in Spark

New Contributor

Thanks @Felix Albani.
I didn't set swap at all for this test. (Removed swap from /etc/fstab and confirmed with 'free').

The settings of /proc/sys/vm/swappiness was 60 which is the default. I tried changing to 10 and 0 as you suggested, but still, I don't see that more RAM is used.
Any other idea?