Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed

avatar
Explorer

We have an end-user who runs queries usually with Impala/HUE.  He is receiving this error, which I assume will require some changes in tuning parameters.  But where do we start to investigate here?  The error message is:

ExecQueryFInstances rpc query_id=e74ef8d9b9215369:4994cbde00000000 failed: Failed to get minimum memory reservation of 68.00 MB on daemon chia1.haas.berkeley.edu:22000 for query e74ef8d9b9215369:4994cbde00000000 because it would exceed an applicable memory limit. Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Memory usage: Process: Limit=97.35 GB Total=80.62 GB Peak=85.72 GB Buffer Pool: Free Buffers: Total=0 Buffer Pool: Clean Pages: Total=4.24 GB Buffer Pool: Unused Reservation: Total=-4.24 GB Free Disk IO Buffers: Total=1.41 GB Peak=1.99 GB RequestPool=root.hue: Total=78.60 GB Peak=82.56 GB Query(2f4b5cff11212907:886aa1400000000): Reservation=77.88 GB ReservationLimit=77.88 GB OtherMemory=731.77 MB Total=78.60 GB Peak=78.92 GB Query(e74ef8d9b9215369:4994cbde00000000): Reservation=0 ReservationLimit=77.88 GB OtherMemory=0 Total=0 Peak=0 RequestPool=root.mdevaan: Total=0 Peak=18.23 GB RequestPool=root.bergquist: Total=0 Peak=84.21 GB RequestPool=root.saqib: Total=0 Peak=8.93 GB Untracked Memory: Total=631.52 MB

Thanks!

1 ACCEPTED SOLUTION

avatar
Super Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
10 REPLIES 10

avatar

 

@Zane- I'm late but can provide some additional insight.

 

I think the suggestion in the error message is a good one (I'm biased because I wrote it, but some thought went into it). "Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error". The general solution for this is to set up admission control with some memory limits so that memory doesn't get oversubscribed, and so that one query can't gobble up more memory than you like. I did a talk at strata that gave pointers on a lot of this things - https://conferences.oreilly.com/strata/strata-ca-2019/public/schedule/detail/73000

 

In this case you can actually see that query 2f4b5cff11212907:886aa1400000000 is using Total=78.60 GB memory, so that's likely your problem.

 

Impala's resource management is totally permissive out of the box and will happily let queries use up all the resources in the system like this.

 

I didn't see what version you're running, but there were a lot of improvements in this area (config options, OOM-avoidance, diagnostics) in CDH6.1+

 

There's various other angles you can take to improve this - if the queries using lots of memory are suboptimal, tuning them (maybe just computing stats) makes a big difference. You can also