Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Getting impala daemon serves via cloudera rest API

Solved Go to solution

Getting impala daemon serves via cloudera rest API

Explorer

Hello,

 

I'm trying to get some key impalad daemon metrics (mem consumption, CPU cycles etc.) from CM REST api but couldn't find it. I'm following this list but only see Impala query level metrics:

 

https://cloudera.github.io/cm_api/apidocs/v15/

 

In the impalad daemon UI, I see lots of such metrics in the /memz and /metrics tab. Are those (or some of them) available from CM REST api as well?

 

Thanks,

S.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Getting impala daemon serves via cloudera rest API

Master Collaborator

If the metrics are available in CM (e.g. if they're in the Impala Charts Library in CM, or you can query them in CM) then you should be able to get them through the CM timeseries API https://cloudera.github.io/cm_api/apidocs/v15/path__timeseries.html

 

CM is capable of collecting all of Impala's metrics from the /metrics page, but it currently only collects a subset of them (the ones we think are most important for cluster monitoring).

 

I'd be interested in knowing if there are useful metrics that are not currently collected by CM, and particularly in understanding the kind of high-level questions that you're trying to answer with metrics, so I can understand if there are metrics that could be added to Impala.

4 REPLIES 4
Highlighted

Re: Getting impala daemon serves via cloudera rest API

Master Collaborator

If the metrics are available in CM (e.g. if they're in the Impala Charts Library in CM, or you can query them in CM) then you should be able to get them through the CM timeseries API https://cloudera.github.io/cm_api/apidocs/v15/path__timeseries.html

 

CM is capable of collecting all of Impala's metrics from the /metrics page, but it currently only collects a subset of them (the ones we think are most important for cluster monitoring).

 

I'd be interested in knowing if there are useful metrics that are not currently collected by CM, and particularly in understanding the kind of high-level questions that you're trying to answer with metrics, so I can understand if there are metrics that could be added to Impala.

Re: Getting impala daemon serves via cloudera rest API

Explorer

Thanks Tim! That's exactly what I want ;-)

 

I believe the metrics listed on this page is what's available from the rest api (I checked a few of them and they are retrievable from the api): https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cm_metrics_impala_daemon.html, it seems to cover all I need as of now.

 

Thanks,

S.

Re: Getting impala daemon serves via cloudera rest API

Master Collaborator

Great to hear! CM's time series support is really pretty powerful.

Re: Getting impala daemon serves via cloudera rest API

Super Collaborator

Hi @Tim Armstrong Hope you are doing well,

 

It will be nice if we have a metirc for the memory part of the daemons_memory_limit used by impala daemon in a given time.

 

So when i get a query failing on memory, i can investigate the memory usage thatwill help me to understand when to increase the limit, secondly, i can learn trend and usage over time and i can plan my increase.

Currently i see only the resident memory per node but this memory isn't used by the queries, so it's a diffcult task for me to investigate the impala behaviour once a query failed on memory.

 

Yes i have a metric of the total memory used by node, but i have different roles in the node, so it hard to track this issue.