question Re: How to monitor resources of impala queue in Archives of Support Questions (Read Only)

How to monitor resources of impala queue

Mobula — Fri, 16 Sep 2022 12:57:37 GMT

Hi guys:

After configured admission control of impala, I met some difficulties.

I tried to monitor the resource usage of the impala queue, but I can't find how to monitor it. Is there any way to monitor the queue of impala like yarn?

After all , impala use fair-scheduler.xml to configure admission control .

Re: How to monitor resources of impala queue

tmarshall — Tue, 13 Mar 2018 22:03:35 GMT

Impala has a webui, by default run on port 25000 of each impalad node, that you may find useful.

In particular, the /queries page will display a list of running or recent queries along with the resource pool they were assigned to

Re: How to monitor resources of impala queue

Mobula — Wed, 14 Mar 2018 01:19:58 GMT

Thanks,

I find that page , but these queries do not show the queue they are on and how much resources are used. I also want to monitor the use of queues, such as how much resources have been used and how much resources are left. Do you have any good way to do this?

Re: How to monitor resources of impala queue

tmarshall — Wed, 14 Mar 2018 21:18:24 GMT

You can see the number of queued/admitted queries and mem used per pool by inspecting the logs, eg. impalad.INFO. Look for lines that come from admission-controller.cc

By default each time a query is admitted we log these stats for the query's pool.

If the default log level doesn't provide enough info, there's additional info that gets logged at higher levels, though of course keep in mind that more logging may slow queries down.

Re: How to monitor resources of impala queue

Mobula — Tue, 20 Mar 2018 15:44:26 GMT

Thanks ! And I find the /metrics page can also show these information which in logs.