Reply
Explorer
Posts: 19
Registered: ‎05-03-2018

File Descriptor limits for Kudu

Hi,

 

We're using CDH 5.12.4.

Kudu user's nofile limit was set initially to 32k. The values were normal for some time, but gradually it raises up.

 

On tables creation the file descriptors significantly raises up, reaching the critical values and causing Kudu instability/failures.

The "Maximum Process File Descriptors" property was raised up twice and currently it seems enough with minimal load for now.

However, later on we're planning to introduce much more load and therefore are very interested in recommended values.

 

Could you share what would be the recommendations for the file descriptors limits for Kudu, e.g. "magic" formulas depending on the load/size/number of nodes.

 

Champion
Posts: 742
Registered: ‎05-16-2016

Re: File Descriptor limits for Kudu

[ Edited ]

could you let us  know how many tablet servers are being used currently and 

also  the Kudu version 

Explorer
Posts: 19
Registered: ‎05-03-2018

Re: File Descriptor limits for Kudu

[ Edited ]

3 Masters

7 Tablet Servers (plans are to double the number of nodes in the nearest future)

 

Kudu comes from CDH: kudu 1.6.0-cdh5.14.2

Champion
Posts: 742
Registered: ‎05-16-2016

Re: File Descriptor limits for Kudu

whats the value on

-block_manager_max_open_files 

 

the default is -1 

Explorer
Posts: 19
Registered: ‎05-03-2018

Re: File Descriptor limits for Kudu

[ Edited ]

This property is left default, i.e.

--block_manager_max_open_files=-1

 The properties are left default, except for 11:

 

- 4 related to dirs (fs_wal_dirs and fs_data_dirs for master and tablet servers)

- Kudu Tablet Server Hard Memory Limit = 20G

- Kudu Tablet Server Block Cache Capacity = 5G

- Automatically Restart Process

- Process Swap Memory Thresholds

- Maximum Process File Descriptors = 65k

- Cgroup CPU Shares=5000

- Cgroup I/O Weight=512

Cloudera Employee
Posts: 60
Registered: ‎04-08-2014

Re: File Descriptor limits for Kudu

[ Edited ]

Please make sure you are not exceeding the scale limits for the version you are running. For Kudu 1.4.0 that can be found at https://kudu.apache.org/releases/1.4.0/docs/known_issues.html

Champion
Posts: 742
Registered: ‎05-16-2016

Re: File Descriptor limits for Kudu

[ Edited ]

Sorry for asking too many questions i am trying to understand the situtation . 

 idealy speaking -1 should be suffice because it uses 40 % of the resource . 

 

 

What does your kudu table look like 

 

 hash and range partitioning ? 
ENCODING Attribute ? 
BLOCK_SIZE ?

 

 

 

Explorer
Posts: 19
Registered: ‎05-03-2018

Re: File Descriptor limits for Kudu

[ Edited ]

"-1" is an option, of course, but it would be good to have better understanding is it normal behaviour, why does it grow and how large might it raise. As well as having some limits (in the case of sudden grow) would be useful to limit the effect on other cluster services.

 

Hash Partitions - majority 50, a couple tables - 128.

Range partitioning unbounded.

Encoding attribute is not used in table creation (auto_encoding).

Block size is not used in table creation (default is taken)

Explorer
Posts: 19
Registered: ‎05-03-2018

Re: File Descriptor limits for Kudu

mpercy, thanks for the link.

Currently the number of tablets per server at some tservers is exceeding recommended up to 10 times.

Others criteria are fine.

 

Does it mean the default limit number of file handlers should be increased 10 times, e.g. from 32k to 320k?

Highlighted
Cloudera Employee
Posts: 60
Registered: ‎04-08-2014

Re: File Descriptor limits for Kudu

[ Edited ]

Hi Andreyeff, it's not recommended to run past the scale limits because it's not tested. Likely you will run into scalability problems due to excessive numbers of threads and context switching, and other things we have not explored yet.

Announcements