Reply
Highlighted
Explorer
Posts: 40
Registered: ‎02-01-2017

Estimated per node memory usage for query much less than actual query memory.

I saw this https://issues.cloudera.org/browse/IMPALA-1832 and I was wondering if there was any update on this.

For example estimated memory used is x and actual memory used per node 3x or 4x. Talking about queries which are using gb of ram. Thanks

Cloudera Employee
Posts: 395
Registered: ‎07-29-2015

Re: Estimated per node memory usage for query much less than actual query memory.

It's expected that the estimates are inaccurate sometimes - getting them totally accurate isn't really possible, since it depends on estimating various runtime values, e.g. how many rows will be produced at different points in the plan.

 

We're actively working on improving the memory management story so that the minimum amount of memory required to run a query is estimated accurately (as opposed to the optimal amount of memory). E.g. https://issues.cloudera.org/browse/IMPALA-4834 and https://issues.cloudera.org/browse/IMPALA-3200. Those are higher priority than improving memory estimates because they'll let us provide stronger guarantees - e.g. if the query is given some minimum amount of memory - xMB, then it will succeed even if we estimated wrongly.

 

Generally we recommend that users set a per-query memory limit (either a default query option or via an admission control pool) to limit resource consumption. If there is a per-query memory limit many operators can spill or scale down there memory consumption.

Announcements