Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

Explorer

I am running CDH 5.2.

 

Can anyone shed any light on the cause of this error?  

 

It seems to me like llama is trying to reserve all of the memory needed to run the query on a single node.  Shouldn't impala spill to disk if it can't get enough memory to run the query purely in memory?

 

2014-12-11 08:29:47,845 WARN com.cloudera.llama.am.LlamaAMServiceImpl: Reserve() error: com.cloudera.llama.util.LlamaException: RESERVATION_ASKING_MORE_MB - Reservation '274f8ef629453f2f:c96063cc39089b9e', expansion 'null' is asking for mo
re memory in mb '64545' than capacity '47747' on node 'analytics-datanode776.eqx-sv5.savagebeast.com'.
com.cloudera.llama.util.LlamaException: RESERVATION_ASKING_MORE_MB - Reservation '274f8ef629453f2f:c96063cc39089b9e', expansion 'null' is asking for more memory in mb '64545' than capacity '47747' on node 'analytics-datanode776.eqx-sv5.sav
agebeast.com'.
at com.cloudera.llama.am.impl.ExpansionReservationsLlamaAM.checkAndUpdateCapacity(ExpansionReservationsLlamaAM.java:170)
at com.cloudera.llama.am.impl.ExpansionReservationsLlamaAM.reserve(ExpansionReservationsLlamaAM.java:129)
at com.cloudera.llama.am.impl.APIContractLlamaAM.reserve(APIContractLlamaAM.java:144)
at com.cloudera.llama.am.LlamaAMServiceImpl.Reserve(LlamaAMServiceImpl.java:132)
at com.cloudera.llama.am.MetricLlamaAMService.Reserve(MetricLlamaAMService.java:140)
at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:512)
at com.cloudera.llama.thrift.LlamaAMService$Processor$Reserve.getResult(LlamaAMService.java:497)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

6 REPLIES 6

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

Explorer

A work around for this problem has been to set the MEM_LIMIT parameter from the impala-shell.  

 

> set mem_limit = 46g;

 

I have the mem_limit set through cloudera manager, but for some reason this property is not being picked up by impala-shell.

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

Master Collaborator

There are two different kinds of mem_limit in Impala. The process mem_limit and a per-query mem_limit.

The one you want to set is the per-query mem_limit which will control how much memory is requested by Impala from Llama.

 

Could you have perhaps set the process mem_limit in CM? That will have no effect on resource requests from Llama.

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

Explorer

The process mem_limit is set to 47G in CM.  However, if I don't set the mem_limit when I run impala-shell the client always requests more memory per node than is avaiable.

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

Master Collaborator

There are known issues with using Llama in conjunction with spilling that we are investigating. Is Llama a requirement in your setup? Until those issues are resolved, you may want to consider admission control instead of Llama.

 

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

Explorer

Llama is a requirement for my setup.  Is there an easy way to have the impala-shell pickup the server side mem_limit configuration?   Since setting mem_limit in the impala-shell seems to allow the queries to run without error every time it is set.

Re: Impala + llama: RESERVATION_ASKING_MORE_MB when running a complex query

Master Collaborator

1. If I understand correctly, what you want is to set a default query mem limit from CM using the safety valve machanism. That should have the effect that any Impala query has a default query mem_limit based on that value (you can still change it on a case by case basis if needed):

 

You can refer to this docs page:
http://www.cloudera.com/content/cloudera/en/documentation/cloudera-impala/latest/topics/impala_confi...

 

Search for 'default_query_option'

 

2. Alternatively, you may want to refer to:

 

http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-1-x/Impala/Installing-and-Using-Im...

 

and consider setting the 'rm_always_use_defaults' option min conjunction with 'rm_default_memory' and 'rm_default_cpu_cores'.

 

Hope it helps.

Don't have an account?
Coming from Hortonworks? Activate your account here