question Recommended value for vm.overcommit_memory for a Spark cluster? in Archives of Support Questions (Read Only)

Recommended value for vm.overcommit_memory for a Spark cluster?

ldaluz — Fri, 13 Nov 2015 13:51:02 GMT

What is the recommended configuration for the vm.overcommit_memory [0|1|2] and vm.overcommit_ratio settings in sysctl?

Looking specifically for a Spark cluster.

I found the following link that suggests vm.overcommit_memory should be set to 1 for mapreduce streaming use cases:

https://www.safaribooksonline.com/library/view/hadoop-operations/9781449327279/ch04.html

Do we have any best practices around this?

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

nsabharwal — Sun, 15 Nov 2015 11:10:40 GMT

@Ram Sriharsha

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

nsabharwal — Thu, 19 Dec 2019 21:53:06 GMT

@Laurence Da Luz

This is very useful https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

and you are right, The best practice is to set vm.overcommit_memory to 1

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

aervits — Tue, 02 Feb 2016 09:46:56 GMT

@Laurence Da Luz can you accept the answer?

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

Augustine — Thu, 19 Dec 2019 20:59:25 GMT

This answer incorrectly summarized the content of the link referenced. The resource listed suggests setting vm.overcommit_memory=1, not vm.overcommit_ratio.

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

lwang — Thu, 19 Dec 2019 21:53:57 GMT

@Augustine ,

Thanks for the feedback. I have corrected the answer and thank you for reporting this.

Cheers,