Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CM enforces max locked memory on datanode process

Solved Go to solution
Highlighted

CM enforces max locked memory on datanode process

New Contributor

Hello,

 

When setting the parameter "dfs.datanode.max.locked.memory" to a value other than 0, it is supposed to be the user's responsibility to set the MEMLOCK limits accordingly. CM does not leave that for the user and enforces it. (as seen on /proc/<pid>/limits)

 

This parameter is used by HDFS only to limt cache resources, so there is no real reason for CM to enforce this limit on an OS level. By doing so, CM does not allow external libraries (non-cache related) on the datanode to allocate memory past this limit. 

 

To make a long story short, I'd like to cancel CM's enforcement of this limit and set the limits myself. Is there any way I could do so?

 

Thanks,

Elad

1 ACCEPTED SOLUTION

Accepted Solutions

Re: CM enforces max locked memory on datanode process

Hi Elad,

You can set probably dfs.datanode.max.locked.memory to the value you want to enforced by the OS, then use the Advanced Configuration Snippet for hdfs-site.xml to change the value that HDFS actually sees for this parameter in hdfs-site.

If CM didn't set memlock, then HDFS caching would be at the mercy of the default memlock setting and could fail unexpectedly, with essentially no workaround. So it's pretty important that CM sets this limit to something that will make caching work.

Thanks,
Darren
2 REPLIES 2

Re: CM enforces max locked memory on datanode process

Hi Elad,

You can set probably dfs.datanode.max.locked.memory to the value you want to enforced by the OS, then use the Advanced Configuration Snippet for hdfs-site.xml to change the value that HDFS actually sees for this parameter in hdfs-site.

If CM didn't set memlock, then HDFS caching would be at the mercy of the default memlock setting and could fail unexpectedly, with essentially no workaround. So it's pretty important that CM sets this limit to something that will make caching work.

Thanks,
Darren

Re: CM enforces max locked memory on datanode process

New Contributor

Thanks a lot Darren, a very simple solution :)

Don't have an account?
Coming from Hortonworks? Activate your account here