File Descriptor limits for Kudu

[#1]

Hi,

We're using CDH 5.12.4.

Kudu user's nofile limit was set initially to 32k. The values were normal for some time, but gradually it raises up.

On tables creation the file descriptors significantly raises up, reaching the critical values and causing Kudu instability/failures.

The "Maximum Process File Descriptors" property was raised up twice and currently it seems enough with minimal load for now.

However, later on we're planning to introduce much more load and therefore are very interested in recommended values.

Could you share what would be the recommendations for the file descriptors limits for Kudu, e.g. "magic" formulas depending on the load/size/number of nodes.

[#2]

could you let us  know how many tablet servers are being used currently and 

also  the Kudu version 

[#3]

3 Masters

7 Tablet Servers (plans are to double the number of nodes in the nearest future)

Kudu comes from CDH: kudu 1.6.0-cdh5.14.2

[#4]

whats the value on

-block_manager_max_open_files 

the default is -1 

[#5]

This property is left default, i.e.

--block_manager_max_open_files=-1

 The properties are left default, except for 11:

- 4 related to dirs (fs_wal_dirs and fs_data_dirs for master and tablet servers)

- Kudu Tablet Server Hard Memory Limit = 20G

- Kudu Tablet Server Block Cache Capacity = 5G

- Automatically Restart Process

- Process Swap Memory Thresholds

- Maximum Process File Descriptors = 65k

- Cgroup CPU Shares=5000

- Cgroup I/O Weight=512

[#6]

Please make sure you are not exceeding the scale limits for the version you are running. For Kudu 1.4.0 that can be found at https://kudu.apache.org/releases/1.4.0/docs/known_issues.html

[#7]

mpercy, thanks for the link.

Currently the number of tablets per server at some tservers is exceeding recommended up to 10 times.

Others criteria are fine.

Does it mean the default limit number of file handlers should be increased 10 times, e.g. from 32k to 320k?

[#8]

Hi Andreyeff, it's not recommended to run past the scale limits because it's not tested. Likely you will run into scalability problems due to excessive numbers of threads and context switching, and other things we have not explored yet.

[#9]

Sorry for asking too many questions i am trying to understand the situtation . 

 idealy speaking -1 should be suffice because it uses 40 % of the resource . 

What does your kudu table look like 

 hash and range partitioning ? 
ENCODING Attribute ? 
BLOCK_SIZE ?

[#10]

"-1" is an option, of course, but it would be good to have better understanding is it normal behaviour, why does it grow and how large might it raise. As well as having some limits (in the case of sudden grow) would be useful to limit the effect on other cluster services.

Hash Partitions - majority 50, a couple tables - 128.

Range partitioning unbounded.

Encoding attribute is not used in table creation (auto_encoding).

Block size is not used in table creation (default is taken)