Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Impala query memory usage

avatar
Explorer

Hey,

I'm investigating memory usage for queries, and noticed to the following parameters (user_name regards to the query initiator):

admission-controller.agg-mem-reserved.root.user_name

- admission-controller.local-backend-mem-reserved.root.user_name

- admission-controller.local-backend-mem-usage.root.user_name

- admission-controller.local-mem-admitted.root.user_name

 

Does someone can give me an explanation for these params? The explanation in impala documentation is not enough.

 

In addition, It will be great if someone could guide me how to monitor the queries actual memory usage - For example - For each node, what is the size in bytes of the query fragment, and the total memory usage in all the cluster for the specific query.

 

Many thanks,

Dror 

1 REPLY 1

avatar

This blog post provides a nice introduction to Impala's admission control:

 

https://blog.cloudera.com/blog/2016/12/resource-management-for-apache-impala-incubating/

 

There are a few ways to inspect a query's memory usage. The query profile and summary will have stats about peak memory usage per host and for each operator in the query. The stats are generally per-host, instead of cluster-wide aggregates. If you're using impala-shell, you can also "set live_summary=1" to get a live update of the query as it makes progress.

 

If you want to see the live state of all queries running on an Impala daemon as an admin, you can look at the "memz" tab of the Impala web UI (by default on port 25000). That will show the full tree of tracked memory from the process down to the operator level.

 

Cloudera Manager also has various charts of aggregate memory usage.