Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Overall peak memory usage of a query?

avatar
New Contributor

Where can I see the overall peak memory usage of a query. In the textplan I get an estimated per-host requirement. But we want to compare RDF queries with up to 200 joins for memory efficiency and need the real per-host peak memory usage. Where can I find it?

2 ACCEPTED SOLUTIONS

avatar

You can find that information in the runtime profile. There are various ways to get it - e.g. from impala-shell you can run profile; after the query. The PerHostPeakMemUsage counter will tell you the peak memory usage for each impala instance executing the query. I think getting the numbers for each host and summing them gives you roughly what you want.

 

200 joins sounds like an interesting query  - let us know how it goes.

View solution in original post

avatar

I misread your question and didn't realise you wanted the per-host peak, PerHostPeakMemUsage gives you exactly what you want.

View solution in original post

2 REPLIES 2

avatar

You can find that information in the runtime profile. There are various ways to get it - e.g. from impala-shell you can run profile; after the query. The PerHostPeakMemUsage counter will tell you the peak memory usage for each impala instance executing the query. I think getting the numbers for each host and summing them gives you roughly what you want.

 

200 joins sounds like an interesting query  - let us know how it goes.

avatar

I misread your question and didn't realise you wanted the per-host peak, PerHostPeakMemUsage gives you exactly what you want.