question Re: How to collect all hadoop metrics from the metrics collector API in Support Questions

How to collect all hadoop metrics from the metrics collector API

umarbiliyamin — Fri, 16 Feb 2018 03:50:20 GMT

I will like to know if there is any way i can collect all the metrics from the metrics collector API using one request with the format below:

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=<>&hostname=<>&appId=<>&startTime=<>&endTime=<>&precision=<>

Thanks

Re: How to collect all hadoop metrics from the metrics collector API

jsensharma — Fri, 16 Feb 2018 06:49:18 GMT

@Biliyaminu Umar

No there is no single call to download (get) all the metrics (services/hosts) from AMS using API. If you want to review the whole AMS data then you might need to review the AMS datastore (HBase). You will have to query them separately. (multiple API calls)

However you can get the list of total metrics that AMS collector is collecting using API metadata call as following: (but this is just metadata about the metrics). Using a single call we can not retrieve all the metrics data at once.

AMS Metadata API: (only metadata)

http://<ams-host>:6188/ws/v1/timeline/metrics/metadata

Re: How to collect all hadoop metrics from the metrics collector API

umarbiliyamin — Fri, 16 Feb 2018 12:46:58 GMT

Hello @Jay Kumar SenSharma

Thanks for your feedback. Please how do i review the AMS datastore. Using this, can i get all the metrics per service?

Thanks

Re: How to collect all hadoop metrics from the metrics collector API

jsensharma — Fri, 16 Feb 2018 13:18:25 GMT

@Biliyaminu Umar

AMS stores the data to HBase tables. So if you can use "sqlline" utility to connect to the AMS HBase database then you should be able to query various HBase tables based on your requirement.

Example: How to connect to AMS database.

# /usr/lib/ambari-metrics-collector/bin/sqlline.py amb25102.example.com:61181:/ams-hbase-unsecure

Please replace "amb25102.example.com" with your Ambari Metrics Collector Hostname and the /ams-hbase-unsecure (with the value that you have for your "zookeeper.znode.parent" mentioned in your Advanced ams-hbase-site)

Once you are able to connect to the DB then you can find the following tables.

Following are the list of tables that you will see

0: jdbc:phoenix:amb25102.example.com:61181:/a> !tables
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
|                TABLE_CAT                 |               TABLE_SCHEM                |                TABLE_NAME                |                TABLE_TYPE                |           |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
|                                          | SYSTEM                                   | CATALOG                                  | SYSTEM TABLE                             |           |
|                                          | SYSTEM                                   | FUNCTION                                 | SYSTEM TABLE                             |           |
|                                          | SYSTEM                                   | SEQUENCE                                 | SYSTEM TABLE                             |           |
|                                          | SYSTEM                                   | STATS                                    | SYSTEM TABLE                             |           |
|                                          |                                          | CONTAINER_METRICS                        | TABLE                                    |           |
|                                          |                                          | HOSTED_APPS_METADATA                     | TABLE                                    |           |
|                                          |                                          | INSTANCE_HOST_METADATA                   | TABLE                                    |           |
|                                          |                                          | METRICS_METADATA                         | TABLE                                    |           |
|                                          |                                          | METRIC_AGGREGATE                         | TABLE                                    |           |
|                                          |                                          | METRIC_AGGREGATE_DAILY                   | TABLE                                    |           |
|                                          |                                          | METRIC_AGGREGATE_HOURLY                  | TABLE                                    |           |
|                                          |                                          | METRIC_AGGREGATE_MINUTE                  | TABLE                                    |           |
|                                          |                                          | METRIC_RECORD                            | TABLE                                    |           |
|                                          |                                          | METRIC_RECORD_DAILY                      | TABLE                                    |           |
|                                          |                                          | METRIC_RECORD_HOURLY                     | TABLE                                    |           |
|                                          |                                          | METRIC_RECORD_MINUTE                     | TABLE                                    |           |
|                                          | ACTIVITY                                 | ACTIVITY_QUEUE                           | TABLE                                    |           |
|                                          | ACTIVITY                                 | CONFIG                                   | TABLE                                    |           |
|                                          | ACTIVITY                                 | COUNTER                                  | TABLE                                    |           |
|                                          | ACTIVITY                                 | FAILED_ACTIVITY                          | TABLE                                    |           |
|                                          | ACTIVITY                                 | HDFS_FILES                               | TABLE                                    |           |
|                                          | ACTIVITY                                 | HDFS_USER_FILE_SUMMARY                   | TABLE                                    |           |
|                                          | ACTIVITY                                 | JOB                                      | TABLE                                    |           |
|                                          | ACTIVITY                                 | RECOMMENDATION                           | TABLE                                    |           |
|                                          | ACTIVITY                                 | SCHEMA_INFO                              | TABLE                                    |           |
|                                          | ACTIVITY                                 | TASK                                     | TABLE                                    |           |
|                                          | ACTIVITY                                 | YARN_APPLICATION                         | TABLE                                    |           |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+

You can query these tables:

0: jdbc:phoenix:amb25102.example.com:61181:/a> select * from SYSTEM.STATS;
0: jdbc:phoenix:amb25102.example.com:61181:/a> select count (*) from METRIC_AGGREGATE;

Please see: https://community.hortonworks.com/articles/71206/using-phoenix-sqlline-utility-to-browse-ambari-met.html

0: jdbc:phoenix:amb25102.example.com:61181:/a> select * from METRIC_AGGREGATE where METRIC_NAME='regionserver.Server.storeFileCount' order by SERVER_TIME limit 1;
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
|               METRIC_NAME                |                  APP_ID                  |               INSTANCE_ID                |               SERVER_TIME                |        UN |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
| regionserver.Server.storeFileCount       | ams-hbase                                |                                          | 1518498780000                            |           |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
1 row selected (0.714 seconds)

Re: How to collect all hadoop metrics from the metrics collector API

umarbiliyamin — Fri, 16 Feb 2018 18:09:37 GMT

@Jay Kumar SenSharma

Oh nice thats very good. Thanks

I have also tried to use the wildcard (%) such that it can return all metrics per service but i always end up with an empty json response. Is it wrong to call for metrics as below;

http://<ams-host>:6188/ws/v1/timeline/metrics?metricNames=regionServer.%

Thanks for your help

Re: How to collect all hadoop metrics from the metrics collector API

jsensharma — Fri, 16 Feb 2018 18:20:44 GMT

@Biliyaminu Umar

Please also specify the startTime / endTime / appId ..etc as well like following:

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=regionserver.Server.%&appId=hbase&startTime=14000000&endTime=14200000

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=regionserver.Server.%&hostname=abc.testdomain124.devlocal&appId=hbase&startTime=14000000&endTime=14200000

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=master.AssignmentManger.ritCount,regionserver.Server.%&hostname=abc.testdomain124.devlocal&appId=hbase&startTime=14000000&endTime=14200000

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=regionserver.Server.%&hostname=abc.testdomain12%.devlocal&appId=hbase&startTime=14000000&endTime=14200000

https://cwiki.apache.org/confluence/display/AMBARI/Metrics+Collector+API+Specification (Section : "Specifying Wild Cards")

Re: How to collect all hadoop metrics from the metrics collector API

umarbiliyamin — Sat, 03 Mar 2018 07:21:36 GMT

Hello @Jay Kumar SenSharma, Thanks for your continous help, it has been insightful. Anyways, i have noticed some behaviour while collecting metrics from the API. If for example i make an API call like so,

http://*********/ws/v1/timeline/metrics?metricNames=dfs.FSNamesystem.BlockCapacity,dfs.FsVolume.FlushIoRateNumOps,jvm.file_descriptor_usage,Topologies,jvm.JvmMetrics.ThreadsTimedWaiting,jvm.threads.runnable.count,jvm.JvmMetrics.GcCount

i get a response which is sorted in a strange manner like this below. Is there a special way API sorts the metrics in the response or is it just a random arrangement. Thanks

{
"metrics": [{
"timestamp": 1520030910000,
"metadata": {},
"metricname": "jvm.JvmMetrics.ThreadsTimedWaiting",
"appid": "applicationhistoryserver",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 29.0
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "Topologies",
"appid": "nimbus",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 0.0
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "dfs.FSNamesystem.BlockCapacity",
"appid": "n



amenode",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 2097152.0
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "dfs.FsVolume.FlushIoRateNumOps",
"appid": "datanode",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 0.0
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "jvm.file_descriptor_usage",
"appid": "nifi",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 0.04996
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "jvm.threads.runnable.count",
"appid": "ambari_server",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 10.0
}
}
]
}

Re: How to collect all hadoop metrics from the metrics collector API

jsensharma — Sat, 03 Mar 2018 07:42:33 GMT

@Biliyaminu Umar

As there are multiple metricsNames there is no common way to sort them out in meaningful/useful form so they are random.

Re: How to collect all hadoop metrics from the metrics collector API

umarbiliyamin — Sat, 03 Mar 2018 07:47:05 GMT

@Jay Kumar SenSharma

Okay thank you very much. I will to find a way of sorting them then.

I appreciate

Re: How to collect all hadoop metrics from the metrics collector API

jsensharma — Sat, 03 Mar 2018 07:50:07 GMT

@Biliyaminu Umar

But if you will have a defined the startTime and endTime for the metrics that you wanted to retrieve then the output should be in the EPOCH time sorted order something like following:

Example:

http://*********//ws/v1/timeline/metrics?metricNames=jvm.JvmMetrics.GcCount._max&hostname=&appId=jobhistoryserver&instanceId=&startTime=1520020359&endTime=1520029247

Then the output will be sorted by time (Epoch time) as following:

{
  "metrics": [
    {
      "timestamp": 1520020500000,
      "metadata": {
        
      },
      "metricname": "jvm.JvmMetrics.GcCount._max",
      "appid": "jobhistoryserver",
      "starttime": 1520020500000,
      "metrics": {
        "1520020500000": 28,
        "1520020800000": 28,
        "1520021100000": 29,
        "1520021400000": 29,
        "1520021700000": 29,
        "1520022000000": 29,
        "1520022300000": 29,
        "1520022600000": 29,
        "1520022900000": 29,
        "1520023200000": 30,
        "1520023500000": 30,
        "1520023800000": 30,
        "1520024100000": 30,
        "1520024400000": 30,
        "1520024700000": 30,
        "1520025000000": 30,
        "1520025300000": 31,
        "1520025600000": 31,
        "1520025900000": 31,
        "1520026200000": 31,
        "1520026500000": 31,
        "1520026800000": 31,
        "1520027100000": 31,
        "1520027400000": 32,
        "1520027700000": 32,
        "1520028000000": 32,
        "1520028300000": 32,
        "1520028600000": 32,
        "1520028900000": 32,
        "1520029200000": 32.333333333333336
      }
    }
  ]
}

Re: How to collect all hadoop metrics from the metrics collector API

jsensharma — Sat, 03 Mar 2018 07:52:02 GMT

@Biliyaminu Umar

Looks like the query asked originally as part of this thread is already answered hence it will be great if you can mark this thread as "Answered" by clicking "Accept" button on the correct answer. That way other HCC users can quickly browse the answered solutions.

Re: How to collect all hadoop metrics from the metrics collector API

umarbiliyamin — Sat, 03 Mar 2018 07:54:38 GMT

Its ok. I am looking at a situation where i collect multiple metrics multiple metrics and extract them for storage in a database.

Thanks anyways

Re: How to collect all hadoop metrics from the metrics collector API

umarbiliyamin — Sat, 03 Mar 2018 08:24:33 GMT

ok, i accepted an answer already.

Thanks