Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala Admission Control and mem_limit usage

Highlighted

Impala Admission Control and mem_limit usage

New Contributor

Hi,

 

according to the documentation when a query is admitted or not depends on the following calculation (with mem_limit enabled) :

  • mem_limit x number_of_total_impala_daemons_in_cluster <= remaining free memory in the pool

Once the query is admitted the remaining free memory is reduced by the previous calculation (mem_limit x number_of_total_impala_daemons). That means that Admission Control keeps this memory reserved during the execution regardless of the real size of the query. 

 

It doesn´t looks very smart because even queries that only need very small resources, takes always the same size from the pool. This is problematics in big clusters where the number of daemons is high.

 

Just to be sure, we have the following questions:

  • The calculation to admit the query is always using the total impala daemons of the cluster or the number of daemons that will be used ?
  • According with your experience. Would be better remove mem_limit and use statistics to calculate the size for admission control  (our queries are very complex with a lot of nested queries) ??

Thanks and best regards.

Don't have an account?
Coming from Hortonworks? Activate your account here