data source for ambari metrics collector

[#1]

I want to get data source ambari metrics collector using for metrics it collects from different servers .

I navigated to path /var/lib/ambari-metrics-collector/hbase/data/default , but i could see many tables out there like METRIC_AGGREGATE, METRIC_AGGREGATE_HOURLY,METRIC_RECORD etc but i wanted to know the data location from where these tables are loading data.

[#2]

@Anurag Mishra

Please try this:

# /usr/lib/ambari-metrics-collector/bin/sqlline.py amb25102.example.com:61181:/ams-hbase-unsecure
0: jdbc:phoenix:amb25102.example.com:61181:/a> !tables
0: jdbc:phoenix:amb25102.example.com:61181:/a> select * from SYSTEM.STATS;
0: jdbc:phoenix:amb25102.example.com:61181:/a> select count (*) from METRIC_AGGREGATE;

.

Please replace "amb25102.example.com" with your Ambari Metrics Collector Hostname and the /ams-hbase-unsecure (with the value that you have for your "zookeeper.znode.parent" mentioned in your Advanced ams-hbase-site)

[#3]

@Jay Kumar SenSharma

yes jay but here we are using phoenix to access over table like selecting count from METRIC_AGGREGATE , but I wanted to get the data source from where it is getting loaded into this table metric_aggregate .

[#4]

@Anurag Mishra
Please check what is the mode of your AMS collectore (Embedded or Distributed)

# grep -B 1 -A 2 'mode' /etc/ambari-metrics-collector/conf/ams-site.xml
    <property>
      <name>timeline.metrics.service.operation.mode</name>
      <value>embedded</value>
    </property><br>

If it is distributed then the Data will be stored on the HDFS.

In case of Embedded Mode the data will be stored in the filesystem something like this.

# grep -B 1 -A 2 'hbase.rootdir' /etc/ambari-metrics-collector/conf/hbase-site.xml
    <property>
      <name>hbase.rootdir</name>
      <value>file:///var/lib/ambari-metrics-collector/hbase</value>
    </property>

# grep -B 1 -A 2 'hbase.rootdir' /etc/ams-hbase/conf/hbase-site.xml
    <property>
      <name>hbase.rootdir</name>
      <value>file:///var/lib/ambari-metrics-collector/hbase</value>
    </property>

Please check the Path which is mentioned for the property "hbase.rootdir" inside your AMS configs "Advanced ams-hbase-site"
Example:

# ls -l /var/lib/ambari-metrics-collector/hbase
total 12
drwxr-xr-x. 2 ams hadoop    6 Feb 15 07:01 archive
drwxr-xr-x. 2 ams hadoop    6 Jan 15 23:24 corrupt
drwxr-xr-x. 4 ams hadoop   32 Aug 11  2017 data
-rw-r--r--. 1 ams hadoop   42 Aug 11  2017 hbase.id
-rw-r--r--. 1 ams hadoop    7 Aug 11  2017 hbase.version
drwxr-xr-x. 2 ams hadoop   43 Feb 15 06:05 MasterProcWALs
drwxr-xr-x. 2 ams hadoop 4096 Feb 15 07:04 oldWALs
drwxr-xr-x. 4 ams hadoop   76 Jan 24 05:49 WALs
[root@amb25102 ~]# grep -B 1 -A 2 'hbase.rootdir' /etc/ambari-metrics-collector/conf/ams-site.xml
[root@amb25102 ~]# 
[root@amb25102 ~]# grep -B 1 -A 2 'hbase.rootdi<br>

.

[#5]

@Jay Kumar SenSharma

Hi Jay

This data would be coming from some log files, actually I needed those log files ,I am not sure but my requirement is to get source data , from where ams is collecting data .

[#6]

@Anurag Mishra

AMS uses HBase internally to store data. So the data files are not plain text which you can read as log files.

Do you not see the following kind of files in your filesystem?

# cd /var/lib/ambari-metrics-collector/hbase

# find . -name "*region*"
./data/hbase/meta/1588230740/.regioninfo
./data/hbase/namespace/ff514c6879eeada03df562a57a5bb206/.regioninfo
./data/default/SYSTEM.CATALOG/ba16524868f30b3ed586ae178ae16168/.regioninfo
./data/default/SYSTEM.SEQUENCE/c7c92a8b77749e975fbdd5f96fac2a18/.regioninfo
./data/default/SYSTEM.SEQUENCE/19e23ba2ed25f11002d1664ca754e25f/.regioninfo
./data/default/SYSTEM.STATS/131dd587a85e8d3f4039b883136f3dea/.regioninfo
./data/default/SYSTEM.FUNCTION/7cd2f7c880cb2ceea902e1e7cb1886c7/.regioninfo
./data/default/METRICS_METADATA/b0eab410b6bd3264c179459502b9deff/.regioninfo
./data/default/HOSTED_APPS_METADATA/e5c8f8d4c4f674cd43d46fe08855376c/.regioninfo
./data/default/INSTANCE_HOST_METADATA/11e0e6f82778ba47074e25d501af699a/.regioninfo
./data/default/CONTAINER_METRICS/e88cc433f3907f7dd7e07c424bd8ed48/.regioninfo
./data/default/METRIC_RECORD/37cb168390e6fa8315f672cd91910657/.regioninfo
./data/default/METRIC_RECORD/a5cd723e901777dfee22430942dd7c1d/.regioninfo
./data/default/METRIC_RECORD_MINUTE/9224d0ceca4410cf5e1c0d4c4cd41f84/.regioninfo
./data/default/METRIC_RECORD_MINUTE/a5975e90a92de88d17ac4532716e906d/.regioninfo
./data/default/METRIC_RECORD_HOURLY/bb252088c2d25c0f95206b1c51c961e8/.regioninfo
./data/default/METRIC_RECORD_DAILY/08e1eef6cce7b95aa5a3cd3384a9c052/.regioninfo
./data/default/METRIC_AGGREGATE/ac6509d99628687bac58d2c2538af08d/.regioninfo
./data/default/METRIC_AGGREGATE/1e635096ee67b68ee8a1affe84fe11eb/.regioninfo
./data/default/METRIC_AGGREGATE_MINUTE/cc816f9b46ec9642c8a7eba87e1e8738/.regioninfo
./data/default/METRIC_AGGREGATE_MINUTE/afa65c4242debbce9af9b6d4f80532e3/.regioninfo
./data/default/METRIC_AGGREGATE_HOURLY/b5a22180a39a0dba20243152324d4d38/.regioninfo
./data/default/METRIC_AGGREGATE_HOURLY/64d933418fbc62bc8dbfcffcd30056cd/.regioninfo
./data/default/METRIC_AGGREGATE_DAILY/032bc9717970f37a3054510bc3212d64/.regioninfo
./data/default/ACTIVITY.SCHEMA_INFO/c51f1da25f56987d80f2d41eb1c54503/.regioninfo
./data/default/ACTIVITY.ACTIVITY_QUEUE/43f9f553453f13a2e546856e41b8a77b/.regioninfo
./data/default/ACTIVITY.FAILED_ACTIVITY/770119a43a30449a5e34aa680f8b029b/.regioninfo
./data/default/ACTIVITY.YARN_APPLICATION/8ed1cbe9be57fe413c98be36cd871e96/.regioninfo
./data/default/ACTIVITY.JOB/d17f35d8aac548168ae040c05f1c1b55/.regioninfo
./data/default/ACTIVITY.TASK/f4e9c2f2fc2e3078619f7425e3747148/.regioninfo
./data/default/ACTIVITY.CONFIG/f1bf9fe26762e7308e373bb9c62d2ecd/.regioninfo
./data/default/ACTIVITY.COUNTER/29488ff5138b549428ee38a586b29a5b/.regioninfo
./data/default/ACTIVITY.RECOMMENDATION/8ff7b2be8bb191350e356a6add453abd/.regioninfo
./data/default/ACTIVITY.HDFS_FILES/dce516f690150a314ac652a333fdbf21/.regioninfo
./data/default/ACTIVITY.HDFS_USER_FILE_SUMMARY/76d9c94c5af03de6d3e3db562e3f7095/.regioninfo
./WALs/hregion-64973741

.

[#7]

@Jay Kumar SenSharma

Yes Jay I can see all these files , but My query is ams would be fetching the data from the cluster and then putting this into hbase,

I wanted to see from where ams fetching all this data and then putting into hbase .

My requirement is to get details like Hbase Dead region server count, Hbase Cluster requests/min Hbase Connections, Hbase CPU Usage, Hbase Memory Usage, Hbase Heap Memory ,Used Hbase Process State ,Hbase Commited memory Hbase Tables Hbase Time in CPU Hbase/Region Servers Online Hbase/Region Server Request per second from the file without using hbase sql .

I was thinking If I could get the data what AMS is using then I would be able to use all this record fro my purpose with use of ams .

so basically I want to get all record without use of ams , for that I wanted files or data from where ams is collecting .

[#8]

@Jay Kumar SenSharma

or in other words If metrics collector would not be there then which file ( logs ) I would be looking to get some metrics ?

[#9]

@Anurag Mishra

As there are multiple related threads with this query opened , Hence putting the following link which answer your query here (Please close the thread if it answers your query)

https://community.hortonworks.com/questions/174953/ambari-rest-api-for-metrics.html

https://community.hortonworks.com/questions/172151/data-source-for-ambari-metrics-collector.html

[#10]

The Ambari Metrics Collector service uses psutil to collect metrics from the host system. This is a python library used for retrieving information on running processes and system utilization. You can read more on psutil within Ambari Metrics Collector here https://github.com/apache/ambari-metrics/tree/master/ambari-metrics-host-monitoring/src/main/python/....

For the various Hadoop sinks, AMS uses the Hadoop sink for collection of the various hadoop components. Source code found here https://github.com/apache/ambari-metrics/blob/master/ambari-metrics-hadoop-sink/src/test/java/org/ap....

AMS uses org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache to store intermediate data until it is time to send it to HBASE. It uses the function emitMetrics to send the Metrics to Hbase whenever the specified amount of time has passed.

I hope this helps you out.