@solardelune This is a great question. The behavior you are observing is actually working exactly as designed by Impala's Admission Controller. It comes down to how the admission controller "clamps" user limits and how the query planner interacts with the MEM_LIMIT parameter. Here is exactly why your query is clamping to 2GiB, why the estimates change, and how to fix it 1. Why your limit was set to 2GiB (The Clamping Behavior) When you configure min-query-mem-limit (2GiB) and max-query-mem-limit (57GiB), you establish a strict memory floor and ceiling for queries running in that specific resource pool You mentioned that you tried setting the limit to 10000000. Because MEM_LIMIT takes values in bytes if no suffix is provided, 10000000 evaluates to just 10 MB By default, Impala has a setting called Clamp MEM_LIMIT Query Option enabled . If a user sets a MEM_LIMIT that falls outside of your configured min/max range, the Admission Controller forcefully "clamps" the limit back into the allowed boundaries . Because your requested 10 MB fell far below the 2GiB minimum, the admission controller clamped it up to your 2GiB floor . Since this was a heavy query, 2GiB was not enough memory to complete the operation, resulting in an Out-Of-Memory (OOM) error. 2. Why the memory estimate changes when you SET MEM_LIMIT It is completely normal that your estimated memory changes when you manually execute SET MEM_LIMIT=4G or 8G When you set an explicit MEM_LIMIT, Impala's query planner takes this hard boundary into account when constructing the execution strategy. Knowing the memory ceiling changes how the planner assigns internal resources—such as when to activate the "spill-to-disk" mechanism or how to size internal memory buffers . Because the underlying query plan adapts to accommodate your new limit, the resulting memory estimate changes as well. In fact, the Impala error logs explicitly warn that "changing the mem_limit may also change the plan" 3. How to make the daemon use the higher memory limits To stop your queries from OOMing and allow them to utilize up to your 57GiB maximum, you have two options: Option A (Manual Override): Explicitly set the MEM_LIMIT for your heavy query to a value between 2GiB and 57GiB using the correct size suffixes (e.g., SET MEM_LIMIT=30G;) . Because 30 GB is comfortably within your boundaries, the Admission Controller will not clamp it, and your query will be granted the 30 GB needed to execute safely Option B (Automatic Estimation via Statistics): If you do not set a MEM_LIMIT, the Admission Controller relies entirely on the planner's memory estimate to decide how much memory to set aside . If your heavy query is only being assigned a 4GB estimate automatically but requires much more, it means your table statistics are missing, stale, or corrupt. You must run the COMPUTE STATS statement on the tables involved in the query . Accurate statistics allow the planner to correctly calculate a much higher memory estimate, which the Admission Controller will then automatically grant within your 2GiB to 57GiB window
... View more
