in HDP 2.6.1 ambari metrics store data in hbase ?

[#1]

Hello All,

We were using following query to get cpu details in hdp 2.3.4 from hbase via poenix, but I do not see any tables or schema for ambari metrics in hdp2.6.1. So do we still have this information in hbase or it is moved to some other components ? Or I am doing something wrong.

select * from METRIC_RECORD where METRIC_NAME like 'cpu%' ;

[#2 · Accepted solution]

@Saurabh

It works in HDP 2.6.1 as well. The AMS collector (Binaries) actually comes from Ambari repo so it basically depends on your Ambari Version.

Example:

# /usr/lib/ambari-metrics-collector/bin/sqlline.py  amb25102.example.com:61181/ams-hbase-unsecure 
0: jdbc:phoenix:amb25102.example.com:61181/am> !tables
0: jdbc:phoenix:amb25102.example.com:61181/am> select * from METRIC_RECORD where METRIC_NAME like 'cpu%' ;

.

Example Output:
================ (From HDP 2.6.1 and Ambari 2.5.1)

0: jdbc:phoenix:amb25102.example.com:61181/am> !tables
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
|                TABLE_CAT                 |               TABLE_SCHEM                |                TABLE_NAME                |                TABLE_TYPE                |                 REMARKS                  |                TY |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
|                                          | SYSTEM                                   | CATALOG                                  | SYSTEM TABLE                             |                                          |                   |
|                                          | SYSTEM                                   | FUNCTION                                 | SYSTEM TABLE                             |                                          |                   |
|                                          | SYSTEM                                   | SEQUENCE                                 | SYSTEM TABLE                             |                                          |                   |
|                                          | SYSTEM                                   | STATS                                    | SYSTEM TABLE                             |                                          |                   |
|                                          |                                          | CONTAINER_METRICS                        | TABLE                                    |                                          |                   |
|                                          |                                          | HOSTED_APPS_METADATA                     | TABLE                                    |                                          |                   |
|                                          |                                          | INSTANCE_HOST_METADATA                   | TABLE                                    |                                          |                   |
|                                          |                                          | METRICS_METADATA                         | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE                         | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE_DAILY                   | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE_HOURLY                  | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE_MINUTE                  | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD                            | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD_DAILY                      | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD_HOURLY                     | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD_MINUTE                     | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | ACTIVITY_QUEUE                           | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | CONFIG                                   | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | COUNTER                                  | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | FAILED_ACTIVITY                          | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | HDFS_FILES                               | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | HDFS_USER_FILE_SUMMARY                   | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | JOB                                      | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | RECOMMENDATION                           | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | SCHEMA_INFO                              | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | TASK                                     | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | YARN_APPLICATION                         | TABLE                                    |                                          |                   |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
0: jdbc:phoenix:amb25102.example.com:61181/am> select * from METRIC_RECORD where METRIC_NAME like 'cpu%' ;
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
|               METRIC_NAME                |                 HOSTNAME                 |               SERVER_TIME                |                  APP_ID                  |               INSTANCE_ID                |                ST |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
| cpu_idle                                 | amb25101.example.com                     | 1510056657199                            | HOST                                     |                                          | 1510056595545     |
| cpu_idle                                 | amb25101.example.com                     | 1510056717578                            | HOST                                     |                                          | 1510056655559     |
| cpu_idle                                 | amb25101.example.com                     | 1510056778195                            | HOST                                     |                                          | 1510056715570     |
| cpu_idle                                 | amb25101.example.com                     | 1510056838608                            | HOST                                     |                                          | 1510056775637     |

.

[#3]

@Saurabh

Regarding your Query "in HDP 2.6.1 ambari metrics store data in hbase ?"

AMS stores data inside HBase.

AMS collector can be configured in two ways "Embedded" (Where it stores data in to local filesystem inside the HBase DB.

In case of "Distributed" mode the Data is stored in HDFS. But HBase is used in both cases.

https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode

.

[#4]

@Saurabh

Following are some very good PPT on Ambari Metrics Collector:

1). Ambari metrics system - Apache ambari meetup (DataWorks Summit 2017)

2). Apache Ambari Meetup - AMS & Grafana

[#5]

Thanks a lot @Jay Kumar SenSharma