Impala query memory usage

[#1]

Hey,

I'm investigating memory usage for queries, and noticed to the following parameters (user_name regards to the query initiator):

- admission-controller.agg-mem-reserved.root.user_name

- admission-controller.local-backend-mem-reserved.root.user_name

- admission-controller.local-backend-mem-usage.root.user_name

- admission-controller.local-mem-admitted.root.user_name

Does someone can give me an explanation for these params? The explanation in impala documentation is not enough.

In addition, It will be great if someone could guide me how to monitor the queries actual memory usage - For example - For each node, what is the size in bytes of the query fragment, and the total memory usage in all the cluster for the specific query.

Many thanks,

Dror 

[#2]

This blog post provides a nice introduction to Impala's admission control:

https://blog.cloudera.com/blog/2016/12/resource-management-for-apache-impala-incubating/

There are a few ways to inspect a query's memory usage. The query profile and summary will have stats about peak memory usage per host and for each operator in the query. The stats are generally per-host, instead of cluster-wide aggregates. If you're using impala-shell, you can also "set live_summary=1" to get a live update of the query as it makes progress.

If you want to see the live state of all queries running on an Impala daemon as an admin, you can look at the "memz" tab of the Impala web UI (by default on port 25000). That will show the full tree of tracked memory from the process down to the operator level.

Cloudera Manager also has various charts of aggregate memory usage.