Overall peak memory usage of a query?

[#1]

Where can I see the overall peak memory usage of a query. In the textplan I get an estimated per-host requirement. But we want to compare RDF queries with up to 200 joins for memory efficiency and need the real per-host peak memory usage. Where can I find it?

[#2 · Accepted solution]

You can find that information in the runtime profile. There are various ways to get it - e.g. from impala-shell you can run profile; after the query. The PerHostPeakMemUsage counter will tell you the peak memory usage for each impala instance executing the query. I think getting the numbers for each host and summing them gives you roughly what you want.

200 joins sounds like an interesting query  - let us know how it goes.

[#3 · Accepted solution]

You can find that information in the runtime profile. There are various ways to get it - e.g. from impala-shell you can run profile; after the query. The PerHostPeakMemUsage counter will tell you the peak memory usage for each impala instance executing the query. I think getting the numbers for each host and summing them gives you roughly what you want.

200 joins sounds like an interesting query  - let us know how it goes.

[#4 · Accepted solution]

I misread your question and didn't realise you wanted the per-host peak, PerHostPeakMemUsage gives you exactly what you want.