need your help about resource logs

[#1]

Hello.

I'm a new guy using Coludera.

I have very simple question about a log.

I need to check every single minute's resource's log such as cpu usage, mem usage, disk I/O etc,

but it's hard to find that kinds of data in log files.

To be more specific, I want to utilize resource logs as data frame shape

for recognizing whether the server is going well or not.

so, can I obtain resource data ?

Aplogies for this stupid question. Appreciate your response.

[#2 · Accepted solution]

Hi @Ben37 ,

Welcome to the Cloudera Community!

Cloudera Manager monitors host resources and stores those in the Host Monitor.

Information is stored via leveldb, so it is not retained in a "log" file as you seeking.  You can find information about how to build charts and query Host and Service Monitor data here:

https://www.cloudera.com/documentation/enterprise/6/6.2/topics/cm_dg_chart_time_series_data.html

The intent of the Host resource metrics provided by Cloudera Manager are intended for use within the context of the cluster itself to monitor the ability of the host to service the needs of the hadoop services.

The information is available via charts in your host, service, and role dashboards.

If you are using Cloudera Manager in your cluster, then it will help to have a look through some of the information here to understand what is available in terms of monitoring cluster health (including host resources):

https://www.cloudera.com/documentation/enterprise/6/6.2/topics/cm_dg_about.html

Cheers,

Ben

[#3]

Hi Ben, 

Appreciate your quick and kind response. 

I have one more question.

After reading your answer, I've started to develope using "time-seriese" api.

During programming, I found a problem with data before about 30 days.

I'd like to obtain a every single minute's data of whole period but, It might be only available every 10 minute's( or longer)  data set

when I tried to get old resource data.

I used this api below.

Could you help me know how to get every single minute's such as, cpu/mem usage ?

import time
import datetime

api_instance = cm_client.TimeSeriesResourceApi(api_client)
from_time = datetime.datetime.fromtimestamp(time.time() - 7776000)
to_time = datetime.datetime.fromtimestamp(time.time())
query = "select cpu_user_rate "\
" where entityname = 'xx' "
# Retrieve time-series data from the Cloudera Manager (CM) time-series data store using a tsquery.
result = api_instance.query_time_series(_from=from_time, query=query, to=to_time)#, desired_rollup='RAW', must_use_desired_rollup = 'true')
ts_list = result.items[0]
for ts in ts_list.time_series:
print (ts.metadata.attributes['entityName'], ts.metadata.metric_name)
for point in ts.data:
print (point.timestamp, point.value)

Appreciate your response.

Ben,