Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

in HDP 2.6.1 ambari metrics store data in hbase ?

avatar
Guru

Hello All,

We were using following query to get cpu details in hdp 2.3.4 from hbase via poenix, but I do not see any tables or schema for ambari metrics in hdp2.6.1. So do we still have this information in hbase or it is moved to some other components ? Or I am doing something wrong.

select * from METRIC_RECORD where METRIC_NAME like 'cpu%' ;

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Saurabh

It works in HDP 2.6.1 as well. The AMS collector (Binaries) actually comes from Ambari repo so it basically depends on your Ambari Version.

Example:

# /usr/lib/ambari-metrics-collector/bin/sqlline.py  amb25102.example.com:61181/ams-hbase-unsecure 
0: jdbc:phoenix:amb25102.example.com:61181/am> !tables
0: jdbc:phoenix:amb25102.example.com:61181/am> select * from METRIC_RECORD where METRIC_NAME like 'cpu%' ;

.

Example Output:
================ (From HDP 2.6.1 and Ambari 2.5.1)

0: jdbc:phoenix:amb25102.example.com:61181/am> !tables
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
|                TABLE_CAT                 |               TABLE_SCHEM                |                TABLE_NAME                |                TABLE_TYPE                |                 REMARKS                  |                TY |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
|                                          | SYSTEM                                   | CATALOG                                  | SYSTEM TABLE                             |                                          |                   |
|                                          | SYSTEM                                   | FUNCTION                                 | SYSTEM TABLE                             |                                          |                   |
|                                          | SYSTEM                                   | SEQUENCE                                 | SYSTEM TABLE                             |                                          |                   |
|                                          | SYSTEM                                   | STATS                                    | SYSTEM TABLE                             |                                          |                   |
|                                          |                                          | CONTAINER_METRICS                        | TABLE                                    |                                          |                   |
|                                          |                                          | HOSTED_APPS_METADATA                     | TABLE                                    |                                          |                   |
|                                          |                                          | INSTANCE_HOST_METADATA                   | TABLE                                    |                                          |                   |
|                                          |                                          | METRICS_METADATA                         | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE                         | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE_DAILY                   | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE_HOURLY                  | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE_MINUTE                  | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD                            | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD_DAILY                      | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD_HOURLY                     | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD_MINUTE                     | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | ACTIVITY_QUEUE                           | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | CONFIG                                   | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | COUNTER                                  | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | FAILED_ACTIVITY                          | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | HDFS_FILES                               | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | HDFS_USER_FILE_SUMMARY                   | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | JOB                                      | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | RECOMMENDATION                           | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | SCHEMA_INFO                              | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | TASK                                     | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | YARN_APPLICATION                         | TABLE                                    |                                          |                   |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
0: jdbc:phoenix:amb25102.example.com:61181/am> select * from METRIC_RECORD where METRIC_NAME like 'cpu%' ;
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
|               METRIC_NAME                |                 HOSTNAME                 |               SERVER_TIME                |                  APP_ID                  |               INSTANCE_ID                |                ST |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
| cpu_idle                                 | amb25101.example.com                     | 1510056657199                            | HOST                                     |                                          | 1510056595545     |
| cpu_idle                                 | amb25101.example.com                     | 1510056717578                            | HOST                                     |                                          | 1510056655559     |
| cpu_idle                                 | amb25101.example.com                     | 1510056778195                            | HOST                                     |                                          | 1510056715570     |
| cpu_idle                                 | amb25101.example.com                     | 1510056838608                            | HOST                                     |                                          | 1510056775637     |

.

View solution in original post

4 REPLIES 4

avatar
Master Mentor

@Saurabh

It works in HDP 2.6.1 as well. The AMS collector (Binaries) actually comes from Ambari repo so it basically depends on your Ambari Version.

Example:

# /usr/lib/ambari-metrics-collector/bin/sqlline.py  amb25102.example.com:61181/ams-hbase-unsecure 
0: jdbc:phoenix:amb25102.example.com:61181/am> !tables
0: jdbc:phoenix:amb25102.example.com:61181/am> select * from METRIC_RECORD where METRIC_NAME like 'cpu%' ;

.

Example Output:
================ (From HDP 2.6.1 and Ambari 2.5.1)

0: jdbc:phoenix:amb25102.example.com:61181/am> !tables
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
|                TABLE_CAT                 |               TABLE_SCHEM                |                TABLE_NAME                |                TABLE_TYPE                |                 REMARKS                  |                TY |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
|                                          | SYSTEM                                   | CATALOG                                  | SYSTEM TABLE                             |                                          |                   |
|                                          | SYSTEM                                   | FUNCTION                                 | SYSTEM TABLE                             |                                          |                   |
|                                          | SYSTEM                                   | SEQUENCE                                 | SYSTEM TABLE                             |                                          |                   |
|                                          | SYSTEM                                   | STATS                                    | SYSTEM TABLE                             |                                          |                   |
|                                          |                                          | CONTAINER_METRICS                        | TABLE                                    |                                          |                   |
|                                          |                                          | HOSTED_APPS_METADATA                     | TABLE                                    |                                          |                   |
|                                          |                                          | INSTANCE_HOST_METADATA                   | TABLE                                    |                                          |                   |
|                                          |                                          | METRICS_METADATA                         | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE                         | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE_DAILY                   | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE_HOURLY                  | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_AGGREGATE_MINUTE                  | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD                            | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD_DAILY                      | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD_HOURLY                     | TABLE                                    |                                          |                   |
|                                          |                                          | METRIC_RECORD_MINUTE                     | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | ACTIVITY_QUEUE                           | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | CONFIG                                   | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | COUNTER                                  | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | FAILED_ACTIVITY                          | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | HDFS_FILES                               | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | HDFS_USER_FILE_SUMMARY                   | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | JOB                                      | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | RECOMMENDATION                           | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | SCHEMA_INFO                              | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | TASK                                     | TABLE                                    |                                          |                   |
|                                          | ACTIVITY                                 | YARN_APPLICATION                         | TABLE                                    |                                          |                   |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
0: jdbc:phoenix:amb25102.example.com:61181/am> select * from METRIC_RECORD where METRIC_NAME like 'cpu%' ;
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
|               METRIC_NAME                |                 HOSTNAME                 |               SERVER_TIME                |                  APP_ID                  |               INSTANCE_ID                |                ST |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+-------------------+
| cpu_idle                                 | amb25101.example.com                     | 1510056657199                            | HOST                                     |                                          | 1510056595545     |
| cpu_idle                                 | amb25101.example.com                     | 1510056717578                            | HOST                                     |                                          | 1510056655559     |
| cpu_idle                                 | amb25101.example.com                     | 1510056778195                            | HOST                                     |                                          | 1510056715570     |
| cpu_idle                                 | amb25101.example.com                     | 1510056838608                            | HOST                                     |                                          | 1510056775637     |

.

avatar
Master Mentor

@Saurabh

Regarding your Query "in HDP 2.6.1 ambari metrics store data in hbase ?"

AMS stores data inside HBase.

AMS collector can be configured in two ways "Embedded" (Where it stores data in to local filesystem inside the HBase DB.

In case of "Distributed" mode the Data is stored in HDFS. But HBase is used in both cases.

https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode

.


avatar
Master Mentor

avatar
Guru

Thanks a lot @Jay Kumar SenSharma