Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Recommended value for vm.overcommit_memory for a Spark cluster?

Solved Go to solution

Recommended value for vm.overcommit_memory for a Spark cluster?

Rising Star

What is the recommended configuration for the vm.overcommit_memory [0|1|2] and vm.overcommit_ratio settings in sysctl?

Looking specifically for a Spark cluster.

I found the following link that suggests vm.overcommit_memory should be set to 1 for mapreduce streaming use cases:

https://www.safaribooksonline.com/library/view/hadoop-operations/9781449327279/ch04.html

Do we have any best practices around this?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

@Laurence Da Luz

This is very useful https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

and you are right, The best practice is to set vm.overcommit_memory to 1

5 REPLIES 5

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

Mentor

@Laurence Da Luz can you accept the answer?

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

@Laurence Da Luz

This is very useful https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

and you are right, The best practice is to set vm.overcommit_memory to 1

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

New Contributor

This answer incorrectly summarized the content of the link referenced.  The resource listed suggests setting vm.overcommit_memory=1, not vm.overcommit_ratio.

Highlighted

Re: Recommended value for vm.overcommit_memory for a Spark cluster?

Community Manager

@Augustine ,

 

Thanks for the feedback. I have corrected the answer and thank you for reporting this.

 

Cheers,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Don't have an account?
Coming from Hortonworks? Activate your account here