- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Impala query memory usage
- Labels:
-
Apache Impala
Created on 01-26-2017 02:59 AM - edited 09-16-2022 03:57 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hey,
I'm investigating memory usage for queries, and noticed to the following parameters (user_name regards to the query initiator):
- admission-controller.agg-mem-reserved.root.user_name
- admission-controller.local-backend-mem-reserved.root.user_name
- admission-controller.local-backend-mem-usage.root.user_name
- admission-controller.local-mem-admitted.root.user_name
Does someone can give me an explanation for these params? The explanation in impala documentation is not enough.
In addition, It will be great if someone could guide me how to monitor the queries actual memory usage - For example - For each node, what is the size in bytes of the query fragment, and the total memory usage in all the cluster for the specific query.
Many thanks,
Dror
Created 01-26-2017 09:53 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This blog post provides a nice introduction to Impala's admission control:
https://blog.cloudera.com/blog/2016/12/resource-management-for-apache-impala-incubating/
There are a few ways to inspect a query's memory usage. The query profile and summary will have stats about peak memory usage per host and for each operator in the query. The stats are generally per-host, instead of cluster-wide aggregates. If you're using impala-shell, you can also "set live_summary=1" to get a live update of the query as it makes progress.
If you want to see the live state of all queries running on an Impala daemon as an admin, you can look at the "memz" tab of the Impala web UI (by default on port 25000). That will show the full tree of tracked memory from the process down to the operator level.
Cloudera Manager also has various charts of aggregate memory usage.
