Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to set ulimit values

avatar
Explorer

Is there any sort of formula or how did you came up with this value for users's processes? is it a random value? what can I check within my cluster in order to get a proper value for me?

 

My current configuration is as of below.

 


# Cloudera modifications
* - nofile 32768
* - nproc 32768

1 REPLY 1

avatar
Super Guru

@Boss ,

 

These are upper bound values to ensure that the services running on the machine won't run into limitations on the number of processes or open file descriptors. IMO, these are really pertinent parameters when you have gateway servers where tens or hundreds users connect to to run their own processes and you want to make sure no single user will run rogue processes that will starve everyone else of resources.

 

The hosts in a CDP cluster enviroment are typically not hosts where users should be connecting directly to. The services and processes that run on those hosts are well known and managed by the administrator. In this scenario, these parameter are not as critical and we usually set them to a value that get them "out of the way", so that that we never reach them.

 

Specifically to answer your question, though:

  • "nofile" is the limit of open file descriptors. Note that file descriptors are not only associated to files; for example, they are also used to refer to open network sockets/ports and pipes. You can check the file descriptors currently open using the command "lsof"
  • "nproc" is the limit of running processes. You can check that with the command "ps".

Cheers,

André

 

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.