How to collect all hadoop metrics from the metrics collector API

[#1]

I will like to know if there is any way i can collect all the metrics from the metrics collector API using one request with the format below:

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=<>&hostname=<>&appId=<>&startTime=<>&endTime=<>&precision=<>

Thanks

[#2]

@Biliyaminu Umar

No there is no single call to download (get) all the metrics (services/hosts) from AMS using API. If you want to review the whole AMS data then you might need to review the AMS datastore (HBase). You will have to query them separately. (multiple API calls)

However you can get the list of total metrics that AMS collector is collecting using API metadata call as following: (but this is just metadata about the metrics). Using a single call we can not retrieve all the metrics data at once.

AMS Metadata API: (only metadata)

http://<ams-host>:6188/ws/v1/timeline/metrics/metadata

.

[#3]

Hello @Jay Kumar SenSharma

Thanks for your feedback. Please how do i review the AMS datastore. Using this, can i get all the metrics per service?

Thanks

[#4]

@Biliyaminu Umar

AMS stores the data to HBase tables. So if you can use "sqlline" utility to connect to the AMS HBase database then you should be able to query various HBase tables based on your requirement.

Example: How to connect to AMS database.

# /usr/lib/ambari-metrics-collector/bin/sqlline.py amb25102.example.com:61181:/ams-hbase-unsecure

Please replace "amb25102.example.com" with your Ambari Metrics Collector Hostname and the /ams-hbase-unsecure (with the value that you have for your "zookeeper.znode.parent" mentioned in your Advanced ams-hbase-site)

.

Once you are able to connect to the DB then you can find the following tables.

Following are the list of tables that you will see

0: jdbc:phoenix:amb25102.example.com:61181:/a> !tables
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
|                TABLE_CAT                 |               TABLE_SCHEM                |                TABLE_NAME                |                TABLE_TYPE                |           |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
|                                          | SYSTEM                                   | CATALOG                                  | SYSTEM TABLE                             |           |
|                                          | SYSTEM                                   | FUNCTION                                 | SYSTEM TABLE                             |           |
|                                          | SYSTEM                                   | SEQUENCE                                 | SYSTEM TABLE                             |           |
|                                          | SYSTEM                                   | STATS                                    | SYSTEM TABLE                             |           |
|                                          |                                          | CONTAINER_METRICS                        | TABLE                                    |           |
|                                          |                                          | HOSTED_APPS_METADATA                     | TABLE                                    |           |
|                                          |                                          | INSTANCE_HOST_METADATA                   | TABLE                                    |           |
|                                          |                                          | METRICS_METADATA                         | TABLE                                    |           |
|                                          |                                          | METRIC_AGGREGATE                         | TABLE                                    |           |
|                                          |                                          | METRIC_AGGREGATE_DAILY                   | TABLE                                    |           |
|                                          |                                          | METRIC_AGGREGATE_HOURLY                  | TABLE                                    |           |
|                                          |                                          | METRIC_AGGREGATE_MINUTE                  | TABLE                                    |           |
|                                          |                                          | METRIC_RECORD                            | TABLE                                    |           |
|                                          |                                          | METRIC_RECORD_DAILY                      | TABLE                                    |           |
|                                          |                                          | METRIC_RECORD_HOURLY                     | TABLE                                    |           |
|                                          |                                          | METRIC_RECORD_MINUTE                     | TABLE                                    |           |
|                                          | ACTIVITY                                 | ACTIVITY_QUEUE                           | TABLE                                    |           |
|                                          | ACTIVITY                                 | CONFIG                                   | TABLE                                    |           |
|                                          | ACTIVITY                                 | COUNTER                                  | TABLE                                    |           |
|                                          | ACTIVITY                                 | FAILED_ACTIVITY                          | TABLE                                    |           |
|                                          | ACTIVITY                                 | HDFS_FILES                               | TABLE                                    |           |
|                                          | ACTIVITY                                 | HDFS_USER_FILE_SUMMARY                   | TABLE                                    |           |
|                                          | ACTIVITY                                 | JOB                                      | TABLE                                    |           |
|                                          | ACTIVITY                                 | RECOMMENDATION                           | TABLE                                    |           |
|                                          | ACTIVITY                                 | SCHEMA_INFO                              | TABLE                                    |           |
|                                          | ACTIVITY                                 | TASK                                     | TABLE                                    |           |
|                                          | ACTIVITY                                 | YARN_APPLICATION                         | TABLE                                    |           |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+

.

You can query these tables:

0: jdbc:phoenix:amb25102.example.com:61181:/a> select * from SYSTEM.STATS;
0: jdbc:phoenix:amb25102.example.com:61181:/a> select count (*) from METRIC_AGGREGATE;

.

Please see: https://community.hortonworks.com/articles/71206/using-phoenix-sqlline-utility-to-browse-ambari-met....

0: jdbc:phoenix:amb25102.example.com:61181:/a> select * from METRIC_AGGREGATE where METRIC_NAME='regionserver.Server.storeFileCount' order by SERVER_TIME limit 1;
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
|               METRIC_NAME                |                  APP_ID                  |               INSTANCE_ID                |               SERVER_TIME                |        UN |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
| regionserver.Server.storeFileCount       | ams-hbase                                |                                          | 1518498780000                            |           |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-----------+
1 row selected (0.714 seconds)

.

[#5]

@Jay Kumar SenSharma

Oh nice thats very good. Thanks

I have also tried to use the wildcard (%) such that it can return all metrics per service but i always end up with an empty json response. Is it wrong to call for metrics as below;

http://<ams-host>:6188/ws/v1/timeline/metrics?metricNames=regionServer.%

Thanks for your help

[#6]

@Biliyaminu Umar

Please also specify the startTime / endTime / appId ..etc as well like following:

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=regionserver.Server.%&appId=hbase&startTim...

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=regionserver.Server.%&hostname=abc.testdom...

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=master.AssignmentManger.ritCount,regionser...

http://<AMS_HOST>:6188/ws/v1/timeline/metrics?metricNames=regionserver.Server.%&hostname=abc.testdom...

.

https://cwiki.apache.org/confluence/display/AMBARI/Metrics+Collector+API+Specification (Section : "Specifying Wild Cards")

[#7]

Hello @Jay Kumar SenSharma, Thanks for your continous help, it has been insightful. Anyways, i have noticed some behaviour while collecting metrics from the API. If for example i make an API call like so,

http://*********/ws/v1/timeline/metrics?metricNames=dfs.FSNamesystem.BlockCapacity,dfs.FsVolume.Flus...

i get a response which is sorted in a strange manner like this below. Is there a special way API sorts the metrics in the response or is it just a random arrangement. Thanks

{
"metrics": [{
"timestamp": 1520030910000,
"metadata": {},
"metricname": "jvm.JvmMetrics.ThreadsTimedWaiting",
"appid": "applicationhistoryserver",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 29.0
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "Topologies",
"appid": "nimbus",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 0.0
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "dfs.FSNamesystem.BlockCapacity",
"appid": "n



amenode",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 2097152.0
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "dfs.FsVolume.FlushIoRateNumOps",
"appid": "datanode",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 0.0
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "jvm.file_descriptor_usage",
"appid": "nifi",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 0.04996
}
}, {
"timestamp": 1520030910000,
"metadata": {},
"metricname": "jvm.threads.runnable.count",
"appid": "ambari_server",
"starttime": 1520030910000,
"metrics": {
"1520030910000": 10.0
}
}
]
}

[#8 · Accepted solution]

@Biliyaminu Umar

As there are multiple metricsNames there is no common way to sort them out in meaningful/useful form so they are random.

[#9]

@Biliyaminu Umar

But if you will have a defined the startTime and endTime for the metrics that you wanted to retrieve then the output should be in the EPOCH time sorted order something like following:

Example:

http://*********//ws/v1/timeline/metrics?metricNames=jvm.JvmMetrics.GcCount._max&hostname=&appId=job...

Then the output will be sorted by time (Epoch time) as following:

{
  "metrics": [
    {
      "timestamp": 1520020500000,
      "metadata": {
        
      },
      "metricname": "jvm.JvmMetrics.GcCount._max",
      "appid": "jobhistoryserver",
      "starttime": 1520020500000,
      "metrics": {
        "1520020500000": 28,
        "1520020800000": 28,
        "1520021100000": 29,
        "1520021400000": 29,
        "1520021700000": 29,
        "1520022000000": 29,
        "1520022300000": 29,
        "1520022600000": 29,
        "1520022900000": 29,
        "1520023200000": 30,
        "1520023500000": 30,
        "1520023800000": 30,
        "1520024100000": 30,
        "1520024400000": 30,
        "1520024700000": 30,
        "1520025000000": 30,
        "1520025300000": 31,
        "1520025600000": 31,
        "1520025900000": 31,
        "1520026200000": 31,
        "1520026500000": 31,
        "1520026800000": 31,
        "1520027100000": 31,
        "1520027400000": 32,
        "1520027700000": 32,
        "1520028000000": 32,
        "1520028300000": 32,
        "1520028600000": 32,
        "1520028900000": 32,
        "1520029200000": 32.333333333333336
      }
    }
  ]
}

.

[#10]

@Biliyaminu Umar

Looks like the query asked originally as part of this thread is already answered hence it will be great if you can mark this thread as "Answered" by clicking "Accept" button on the correct answer. That way other HCC users can quickly browse the answered solutions.