How can I get some monitoring data from cloudera manager(CM)?

[#1]

If I want to get some monitor data from the CM, do I need to use tsquery? How can I user tsquery？ Can you give me an example of using tsquery?

[#2 · Accepted solution]

@littlewolf you should try to use the CM API, it has a Java api.

I don't know the specific metric you are looking for but it may be there. I have used these to gather all queries issues to Hive and Impala to due some usage analysis. If it isn't in CM I would look at collecting it from the source. Most of the services and components expose metrics through JMX and host stats can be collected through some of the typical methods.

https://cloudera.github.io/cm_api/

[#3]

@littlewolf

Syntax: SELECT [metric expression] WHERE [predicate]

To start with tsquery, you can click on any existing charts in CM, it will show you the output based on tsquery. In general, it won't allow you to alter the existing chart and it is not recommended. So you can go to CM -> Charts -> Chart builder -> follow the above syntax, it will automatically pop-up metric expression and predicate.

Ex: SELECT physical_memory_used 

https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cm_dg_tsquery.html

[#4]

If I want to call these data in my program, can you give me an example of calling with Java API? You can use this physical_memory example，how can I get this data in my program. I'm a new hand and can give me detailed example about this? Thank You very much.

[#5]

@littlewolf

Pls refer the below links for time series queries, it may help you (i am not a java resource)

https://github.com/jrachiele/java-timeseries

https://github.com/jrachiele/java-timeseries/blob/master/src/test/java/timeseries/models/MeanModelSp...

[#6 · Accepted solution]

@littlewolf you should try to use the CM API, it has a Java api.

I don't know the specific metric you are looking for but it may be there. I have used these to gather all queries issues to Hive and Impala to due some usage analysis. If it isn't in CM I would look at collecting it from the source. Most of the services and components expose metrics through JMX and host stats can be collected through some of the typical methods.

https://cloudera.github.io/cm_api/

[#7]

@mbigelow Do you collect the metrics using a code or a script?

i'm collecting alot of metrics from Yarn histroy server API and the impala API, intersting in the specific metrics you collected regarding the hive and impala usage analytics

[#8 · Accepted solution]

@Fawze I don't collect specific metric, yet. I make an api call to get all Hive jobs between this time and that time (same for Impala) from... This data is then crunched to provide usage analysis for these specific types of jobs.

/clusters/{clusterName}/services/{serviceName}/yarnApplications
/clusters/{clusterName}/services/{serviceName}/impalaQueries