Reply
rdh
Explorer
Posts: 13
Registered: ‎11-25-2014

Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

I am running CDH 5.2.

 

Can anyone shed any light on the cause of this error?  

 

It seems to me like llama is trying to reserve all of the memory needed to run the query on a single node.  Shouldn't impala spill to disk if it can't get enough memory to run the query purely in memory?

 

2014-12-11 08:29:47,845 WARN com.cloudera.llama.am.LlamaAMServiceImpl: Reserve() error: com.cloudera.llama.util.LlamaException: RESERVATION_ASKING_MORE_MB - Reservation '274f8ef629453f2f:c96063cc39089b9e', expansion 'null' is asking for mo
re memory in mb '64545' than capacity '47747' on node 'analytics-datanode776.eqx-sv5.savagebeast.com'.
com.cloudera.llama.util.LlamaException: RESERVATION_ASKING_MORE_MB - Reservation '274f8ef629453f2f:c96063cc39089b9e', expansion 'null' is asking for more memory in mb '64545' than capacity '47747' on node 'analytics-datanode776.eqx-sv5.sav
agebeast.com'.
at com.cloudera.llama.am.impl.ExpansionReservationsLlamaAM.checkAndUpdateCapacity(ExpansionReservationsLlamaAM.java:170)
at com.cloudera.llama.am.impl.ExpansionReservationsLlamaAM.reserve(ExpansionReservationsLlamaAM.java:129)
at com.cloudera.llama.am.impl.APIContractLlamaAM.reserve(APIContractLlamaAM.java:144)
at com.cloudera.llama.am.LlamaAMServiceImpl.Reserve(LlamaAMServiceImpl.java:132)
at com.cloudera.llama.am.MetricLlamaAMService.Reserve(MetricLlamaAMService.java:140)
at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:512)
at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:497)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

rdh
Explorer
Posts: 13
Registered: ‎11-25-2014

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

A work around for this problem has been to set the MEM_LIMIT parameter from the impala-shell.  

 

> set mem_limit = 46g;

 

I have the mem_limit set through cloudera manager, but for some reason this property is not being picked up by impala-shell.

Cloudera Employee
Posts: 307
Registered: ‎10-16-2013

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

There are two different kinds of mem_limit in Impala. The process mem_limit and a per-query mem_limit.

The one you want to set is the per-query mem_limit which will control how much memory is requested by Impala from Llama.

 

Could you have perhaps set the process mem_limit in CM? That will have no effect on resource requests from Llama.

rdh
Explorer
Posts: 13
Registered: ‎11-25-2014

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

The process mem_limit is set to 47G in CM.  However, if I don't set the mem_limit when I run impala-shell the client always requests more memory per node than is avaiable.

Cloudera Employee
Posts: 307
Registered: ‎10-16-2013

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

There are known issues with using Llama in conjunction with spilling that we are investigating. Is Llama a requirement in your setup? Until those issues are resolved, you may want to consider admission control instead of Llama.

 

rdh
Explorer
Posts: 13
Registered: ‎11-25-2014

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

Llama is a requirement for my setup.  Is there an easy way to have the impala-shell pickup the server side mem_limit configuration?   Since setting mem_limit in the impala-shell seems to allow the queries to run without error every time it is set.

Cloudera Employee
Posts: 307
Registered: ‎10-16-2013

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

1. If I understand correctly, what you want is to set a default query mem limit from CM using the safety valve machanism. That should have the effect that any Impala query has a default query mem_limit based on that value (you can still change it on a case by case basis if needed):

 

You can refer to this docs page:
http://www.cloudera.com/content/cloudera/en/documentation/cloudera-impala/latest/topics/impala_confi...

 

Search for 'default_query_option'

 

2. Alternatively, you may want to refer to:

 

http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-1-x/Impala/Installing-and-Using-Im...

 

and consider setting the 'rm_always_use_defaults' option min conjunction with 'rm_default_memory' and 'rm_default_cpu_cores'.

 

Hope it helps.