question Overall peak memory usage of a query? in Support Questions

Overall peak memory usage of a query?

manuelschneid3r — Fri, 18 Dec 2015 11:55:07 GMT

Where can I see the overall peak memory usage of a query. In the textplan I get an estimated per-host requirement. But we want to compare RDF queries with up to 200 joins for memory efficiency and need the real per-host peak memory usage. Where can I find it?

Re: Overall peak memory usage of a query?

Tim Armstrong — Fri, 18 Dec 2015 17:16:52 GMT

You can find that information in the runtime profile. There are various ways to get it - e.g. from impala-shell you can run profile; after the query. The PerHostPeakMemUsage counter will tell you the peak memory usage for each impala instance executing the query. I think getting the numbers for each host and summing them gives you roughly what you want.

200 joins sounds like an interesting query - let us know how it goes.

Re: Overall peak memory usage of a query?

Tim Armstrong — Fri, 18 Dec 2015 17:17:35 GMT

I misread your question and didn't realise you wanted the per-host peak, PerHostPeakMemUsage gives you exactly what you want.