Skip to main content
Ben37
Ben37
Silver ⭐️⭐️
June 3, 2019
Solved

need your help about resource logs

Hello.

 

I'm a new guy using Coludera.

I have very simple question about a log.

 

I need to check every single minute's resource's log such as cpu usage, mem usage, disk I/O etc,

but it's hard to find that kinds of data in log files.

To be more specific, I want to utilize resource logs as data frame shape

for recognizing whether the server is going well or not.

 

so, can I obtain resource data ?

 

Aplogies for this stupid question. Appreciate your response.

 

 

bgooley
bgooleyAnswer
Gold ⭐️
June 3, 2019

Hi @Ben37 ,

 

Welcome to the Cloudera Community!

 

Cloudera Manager monitors host resources and stores those in the Host Monitor.

Information is stored via leveldb, so it is not retained in a "log" file as you seeking.  You can find information about how to build charts and query Host and Service Monitor data here:

 

https://www.cloudera.com/documentation/enterprise/6/6.2/topics/cm_dg_chart_time_series_data.html

 

The intent of the Host resource metrics provided by Cloudera Manager are intended for use within the context of the cluster itself to monitor the ability of the host to service the needs of the hadoop services.

 

The information is available via charts in your host, service, and role dashboards.

 

If you are using Cloudera Manager in your cluster, then it will help to have a look through some of the information here to understand what is available in terms of monitoring cluster health (including host resources):

 

https://www.cloudera.com/documentation/enterprise/6/6.2/topics/cm_dg_about.html

 

Cheers,

 

Ben

View in thread

2 replies

bgooley
Gold ⭐️
bgooleyAnswer
Gold ⭐️
June 3, 2019

Hi @Ben37 ,

 

Welcome to the Cloudera Community!

 

Cloudera Manager monitors host resources and stores those in the Host Monitor.

Information is stored via leveldb, so it is not retained in a "log" file as you seeking.  You can find information about how to build charts and query Host and Service Monitor data here:

 

https://www.cloudera.com/documentation/enterprise/6/6.2/topics/cm_dg_chart_time_series_data.html

 

The intent of the Host resource metrics provided by Cloudera Manager are intended for use within the context of the cluster itself to monitor the ability of the host to service the needs of the hadoop services.

 

The information is available via charts in your host, service, and role dashboards.

 

If you are using Cloudera Manager in your cluster, then it will help to have a look through some of the information here to understand what is available in terms of monitoring cluster health (including host resources):

 

https://www.cloudera.com/documentation/enterprise/6/6.2/topics/cm_dg_about.html

 

Cheers,

 

Ben

Translate
    Ben37
    Ben37Author
    Silver ⭐️⭐️
    June 4, 2019

    Hi Ben, 

     

    Appreciate your quick and kind response. 

     

    I have one more question.

    After reading your answer, I've started to develope using "time-seriese" api.

    During programming, I found a problem with data before about 30 days.

    I'd like to obtain a every single minute's data of whole period but, It might be only available every 10 minute's( or longer)  data set

    when I tried to get old resource data.

     

    I used this api below.

    Could you help me know how to get every single minute's such as, cpu/mem usage ?

     

    import time
    import datetime

    api_instance = cm_client.TimeSeriesResourceApi(api_client)
    from_time = datetime.datetime.fromtimestamp(time.time() - 7776000)
    to_time = datetime.datetime.fromtimestamp(time.time())
    query = "select cpu_user_rate "\
    " where entityname = 'xx' "
    # Retrieve time-series data from the Cloudera Manager (CM) time-series data store using a tsquery.
    result = api_instance.query_time_series(_from=from_time, query=query, to=to_time)#, desired_rollup='RAW', must_use_desired_rollup = 'true')
    ts_list = result.items[0]
    for ts in ts_list.time_series:
    print (ts.metadata.attributes['entityName'], ts.metadata.metric_name)
    for point in ts.data:
    print (point.timestamp, point.value)

     

    Appreciate your response.

    Ben,

    Translate
    Terms & ConditionsAccessibility statement