question Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed in Support Questions

ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

Zane- — Tue, 21 Apr 2026 11:37:02 GMT

We have an end-user who runs queries usually with Impala/HUE. He is receiving this error, which I assume will require some changes in tuning parameters. But where do we start to investigate here? The error message is:

ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed: Failed to get minimum memory reservation of 68.00 MB on daemon chia1.haas.berkeley.edu:22000 for query e74ef8d9b9215369:4994cbde00000000 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage: Process: Limit=97.35 GB Total=80.62 GB Peak=85.72 GB Buffer Pool: Free Buffers: Total=0 Buffer Pool: Clean Pages: Total=4.24 GB Buffer Pool: Unused Reservation: Total=-4.24 GB Free Disk IO Buffers: Total=1.41 GB Peak=1.99 GB RequestPool=root.hue: Total=78.60 GB Peak=82.56 GB Query(2f4b5cff11212907:886aa1400000000): Reservation=77.88 GB ReservationLimit=77.88 GB OtherMemory=731.77 MB Total=78.60 GB Peak=78.92 GB Query(e74ef8d9b9215369:4994cbde00000000): Reservation=0 ReservationLimit=77.88 GB OtherMemory=0 Total=0 Peak=0 RequestPool=root.mdevaan: Total=0 Peak=18.23 GB RequestPool=root.bergquist: Total=0 Peak=84.21 GB RequestPool=root.saqib: Total=0 Peak=8.93 GB Untracked Memory: Total=631.52 MB

Thanks!

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

AcharkiMed — Thu, 08 Aug 2019 07:12:55 GMT

Hi @Zane-

This is an OOM (Out If Memory) error, it simply means that this query needs more memory to be completed; it usually occurs when the cluster is in charge. The request has exceeded the existing memory.
The solution is to add more memory to your nodes or to add more nodes. Otherwise, try to optimize your queries, and you can make sure that your impala setting is optimal too.

https://www.cloudera.com/documentation/enterprise/latest/topics/impala_performance.html

Good Luck.

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

EricL — Fri, 09 Aug 2019 08:36:12 GMT

Hi Zane,

Please share the failed query PROFILE as attachment to the thread if possible.

Also, what's the default MEMORY_LIMIT for Impala? Have you tried to increase this limit to see if it can help with the query?

SET MEMORY_LIMIT=20g;

Of course the value will depend on how much memory you have on your impala daemon host.

Cheers
Eric

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

Zane- — Thu, 29 Aug 2019 16:36:01 GMT

While I would normally assume this is true, the error is occurring on queries that were previously successful and completing without errors. Something has changed the memory consumption behavior and the error message received hasn't been particularly helpful for troubleshooting. I will continue the investigation.

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

Zane- — Thu, 29 Aug 2019 19:02:40 GMT

I increased the MEMORY_LIMIT to 96GB (up from ~79GB), and I cannot increase it any further without over-allocating memory overall.

The latest query errors have been similar to this one, whose profile I have attached.

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

Zane- — Thu, 29 Aug 2019 19:48:13 GMT

Here is another query profile, which the error condition just occurred for the end-user, attached.

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

EricL — Thu, 29 Aug 2019 23:22:17 GMT

@Zane,

The issues I can see from this profile as below:

1. The query got closed due to client inactivity:
Query f64da6a899944136:22dcac6800000000 expired due to client inactivity (timeout is 10m)

Not because of OOM

2. you do not have table stats, which can cause impala to use bad query plan:

WARNING: The following tables are missing relevant table and/or column statistics.
external_data.physician_compare

This is evident that in the SUMMARY section, the estimated rows are -1:

Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
--------------------------------------------------------------------------------------------------------------------------
01:EXCHANGE 1 189.890ms 189.890ms 1.02K -1 0 0 UNPARTITIONED
00:SCAN HDFS 5 98.859ms 104.753ms 22.53K -1 78.33 MB 176.00 MB external_data.physician_compare

3. Query finished in about 1 second, as result was ready after 1.45 seconds:

First row fetched: 1.45s

4. I can see the query is most likely from hue:

Session Type: HIVESERVER2

So my conclusion is that user stayed in Hue Hive editor idle until query is closed from server side. There is no issue from the query itself whatsoever.

Cheers

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

EricL — Thu, 29 Aug 2019 23:25:36 GMT

It is same as another profile, please see my response here:
https://community.cloudera.com/t5/Support-Questions/ExecQueryFInstances-rpc-query-id-e74ef8d9b9215369/m-p/269400/highlight/true#M206803

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

Zane- — Fri, 30 Aug 2019 17:30:38 GMT

Thank you for your assessment, which is spot-on.

I think that the adjustments to MEMORY_LIMIT plus previously setting the idle session and query limits to 10m helped with the prior OOM problem, especially when multiple users are active. We are relatively new to Hadoop/Impala and so we are still learning a lot about how to configure and work with the data within the environment, in tuning the performance, and tailoring the configuration to work well within the limits of the available hardware.

In general, I wish there were documentation that specifically address errors and performance tuning. It would provide more insights into troubleshooting problems and interpreting the errors received. Maybe there is a book already published that's good for this? It would be a huge help!

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

EricL — Sat, 31 Aug 2019 00:50:59 GMT

@Zane-

There is some troubleshooting guide in the link below:
https://www.cloudera.com/documentation/enterprise/latest/topics/impala_troubleshooting.html

which can help a bit. I have also wrote some impala profile tutorial, which can be found below:

https://www.ericlin.me/2018/09/impala-query-profile-explained-part-1/
https://www.ericlin.me/2019/04/impala-query-profile-explained-part-2/
https://www.ericlin.me/2019/05/impala-query-profile-explained-part-3/

Hope they can be helpful.

Cheers
Eric

Re: ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

Tim Armstrong — Fri, 20 Sep 2019 17:04:59 GMT

@Zane- I'm late but can provide some additional insight.

I think the suggestion in the error message is a good one (I'm biased because I wrote it, but some thought went into it). "Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error". The general solution for this is to set up admission control with some memory limits so that memory doesn't get oversubscribed, and so that one query can't gobble up more memory than you like. I did a talk at strata that gave pointers on a lot of this things - https://conferences.oreilly.com/strata/strata-ca-2019/public/schedule/detail/73000

In this case you can actually see that query 2f4b5cff11212907:886aa1400000000 is using Total=78.60 GB memory, so that's likely your problem.

Impala's resource management is totally permissive out of the box and will happily let queries use up all the resources in the system like this.

I didn't see what version you're running, but there were a lot of improvements in this area (config options, OOM-avoidance, diagnostics) in CDH6.1+

There's various other angles you can take to improve this - if the queries using lots of memory are suboptimal, tuning them (maybe just computing stats) makes a big difference. You can also