Recommended value for vm.overcommit_memory for a Spark cluster?

[#1]

What is the recommended configuration for the vm.overcommit_memory [0|1|2] and vm.overcommit_ratio settings in sysctl?

Looking specifically for a Spark cluster.

I found the following link that suggests vm.overcommit_memory should be set to 1 for mapreduce streaming use cases:

https://www.safaribooksonline.com/library/view/hadoop-operations/9781449327279/ch04.html

Do we have any best practices around this?

[#2]

@Ram Sriharsha

[#3]

@Laurence Da Luz can you accept the answer?

[#4 · Accepted solution]

@Laurence Da Luz

This is very useful https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

and you are right, The best practice is to set vm.overcommit_memory to 1

[#5]

This answer incorrectly summarized the content of the link referenced.  The resource listed suggests setting vm.overcommit_memory=1, not vm.overcommit_ratio.

[#6]

@Augustine ,

Thanks for the feedback. I have corrected the answer and thank you for reporting this.

Cheers,

Li