Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Highlighted

Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Rising Star

Hi,

Trying to play around with ambari metrics on HDP 2.6 sandbox, but still cannot figure out many details...

I'm following this tutorial : http://bryanbende.com/development/2015/07/31/ambari-metrics-part1-metrics-collector

I tried to send a fake_metric by issuing following command :

curl -H "Content-Type: application/json" -X POST -d '{"metrics": 
[{"metricname": "AMBARI_METRICS.SmokeTest.FakeMetric", "appid": 
"amssmoketestfake", "hostname": "ambari20-5.c.pramod-thangali.internal",
 "timestamp": 1432075898000, "starttime": 1432075898000, "metrics": 
{"1432075898000": 0.963781711428, "1432075899000": 1432075898000}}]}' 
http://127.0.0.1:6188/ws/v1/timeline/metrics

First, wondering which host/port I should try to target :

I guess host is 127.0.0.1 as I ssh on the sandbox itself (single VM), but which port should I use : the one specified in "timeline.metrics.service.webapp.address" property ?

Anyway, It returns a successfull HTTP 200 code, so I suppose everything went fine.

The question now is "how to check that the metric has been properly received/processed/stored ?"

First, I tried to retrieve it with following command :

curl -vvv -X GET 
http://127.0.0.1:6188/ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric&appId=a...

This returns once again a HTTP 200 status code, and following data : {"metrics":[]}

=> Why doesn't it return what I sent in the first command ?

Then I decided to look at phoenix DB to check its content, but faced another issue : When I tried following command :

./sqlline.py localhost:61181:/hbase

It simply failed with NPE :

...

Caused by: java.lang.NullPointerException
  at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:463)
  at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
  at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
  at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1186)

...

I tried to figure out once again which port & znode to use :

In ambari configuration, I noticed that :

  • "hbase.zookeeper.property.clientPort" property had "{{zookeeper_clientPort}}" value. Not sure about it but I searched for this value in ams-site.xml configuration file and it turned out that "cluster.zookeeper.property.clientPort" property had "2181" value.
  • "zookeeper.znode.parent" property had "/ams-hbase-unsecure" value

So, I tried again with following command :

./sqlline.py localhost:2181:/ams-hbase-unsecure

And still got the same exception...

What's wrong with my configuration ?

Thanks for any help

9 REPLIES 9
Highlighted

Re: Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Super Mentor

@Sebastien Chausson

In your curl command looks like you are using incorrect host name from which yu want to retrieve the Metrics, that you will need to replace with your sandbox "# hostname -f" command output.

# hostname -f 

Notice: amssmoketestfake&hostname=ambari20-5.c.pramod-thangali.internal

You should change the mentioned hostname in both POST & GET requests to your sandbox hostname as following:

"hostname": "sandbox.hortonworks.com"

Also while retrieving metrics from Embedded Mode AMS you should try to use: Login to sandbox using ssh on port 2222

Example:

# ssh root@localhost -p 2222

Then try

[root@sandbox ~]# hostname -f
sandbox.hortonworks.com

[root@sandbox ~]# 
[root@sandbox ~]# /usr/hdp/2.6.0.3-8/phoenix/bin/sqlline.py sandbox.hortonworks.com:61181:/ams-hbase-unsecure

.

Highlighted

Re: Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Rising Star

Hi Jay,

Thanks for your help. Second part of your answer was very helpful as it permitted me to check that nothing had been stored in phoenix table. I tried to POST a new metric with correct hostname as you suggested :

curl -H "Content-Type: application/json" -X POST -d '{"metrics": 
[{"metricname": "AMBARI_METRICS.SmokeTest.FakeMetric", "appid": 
"amssmoketestfake", "hostname": "sandbox.hortonworks.com", "timestamp": 
1432075898000, "starttime": 1432075898000, "metrics": {"1432075898000": 
0.963781711428, "1432075899000": 1432075898000}}]}' <a href="http://127.0.0.1:6188/ws/v1/timeline/metrics">http://127.0.0.1:6188/ws/v1/timeline/metrics</a>

But again when I tried to retrieve this metrics, following returned empty data :

curl -X GET 
http://127.0.0.1:6188/ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric&appId=a... precision=seconds& startTime=1432075838000&endTime=1432075959000

Is there a way to check on AMS side (log files...?) if it properly received the metric I sent ?

Highlighted

Re: Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Super Mentor

@Sebastien Chausson

Wow!!! good to know that the initial issue is resolved.

I will need to try the GET part on my own. But looks like the following part if causing the issue here:

≺ision=seconds☆tTime

Highlighted

Re: Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Rising Star

oups, copy/paste issue, I updated the GET command above with the exact one I submitted without any success...

Highlighted

Re: Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Super Mentor

@Sebastien Chausson

Please refer to the following link:

https://cwiki.apache.org/confluence/display/AMBARI/Metrics+Collector+API+Specification

Then try using the following API calls inside your Sandbox Terminal to POST the dummy metrics.

# curl -H "Content-Type: application/json" -X POST -d '{"metrics": [{"metricname": "AMBARI_METRICS.SmokeTest.FakeMetric", "appid": "amssmoketestfake", "hostname": "sandbox.hortonworks.com", "timestamp": 1432075898000, "starttime": 1432075898000, "metrics": {"1432075898000": 0.963781711428, "1432075899000": 1432075898000}}]}' "http://sandbox.hortonworks.com:6188/ws/v1/timeline/metrics"

.

Use the following GET call to fetch the dummy metrics data:

# curl -H "Content-Type: application/json" -X GET "http://sandbox.hortonworks.com:6188/ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric&appId=amssmoketestfake&hostname=sandbox.hortonworks.com"

-

Sample Output :

[root@sandbox ~]# curl -H "Content-Type: application/json" -X GET "http://sandbox.hortonworks.com:6188/ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric&appId=amssmoketestfake&hostname=sandbox.hortonworks.com"


{"metrics":[{"timestamp":1497688698786,"metadata":{},"metricname":"AMBARI_METRICS.SmokeTest.FakeMetric","appid":"amssmoketestfake","hostname":"sandbox.hortonworks.com","starttime":1497688690000,"metrics":{"1497688691000":1.49768869E12}}]}

.

Highlighted

Re: Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Rising Star

@Jay SenSharma

Hi Jay,

I ran your sample requests successfully, but I'm wondering if this is possible to get all the values of the dummy metric with the REST API. In your example, you post 2 values for your dummy metric but the GET request only returns the latest value. Documentation states that ambari collector only returns latest value if startTime & endTime are not specified in the request, but even by adding thoses 2 parameters, I couldn't get the 2 original values...Any idea ?

Thanks again for your help

Highlighted

Re: Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Rising Star

@Jay SenSharma

I just gave a try to your suggestion, but still cannot get the same result as you :

1 - POST dummy metric :

curl -H "Content-Type: application/json" -X POST -d '{"metrics": [{"metricname": "AMBARI_METRICS.SmokeTest.FakeMetric", "appid": "amssmoketestfake", "hostname": "sandbox.hortonworks.com", "timestamp": 1432075898000, "starttime": 1432075898000, "metrics": {"1432075898000": 0.963781711428, "1432075899000": 1432075898000}}]}' "http://sandbox.hortonworks.com:6188/ws/v1/timeline/metrics"

Output :

{"errors":[]}

2 - GET dummy metrics :

curl -H "Content-Type: application/json" -X GET "http://sandbox.hortonworks.com:6188/ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric&appId=amssmoketestfake&hostname=sandbox.hortonworks.com"

Output :

{"metrics":[]}

Cannot understand why posted metrics don't show up here...

Highlighted

Re: Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Just a couple of days ago I was using my own blog post to debug a metrics issue on a HDF install and I needed to do two things to get it to work:

- The connect string I needed to use was

./bin/sqlline.py localhost:61181:/ams-hbase-unsecure

- I needed to use an older version of the phoenix client which I had to find in Apache archives, I can't remember exactly which version, but I want to say I used 4.4 or 4.2... whenever I used the latest ones like 4.10, 4.9. 4.8, I was getting that exception. Even with the older one there was an exception that printed, but it still connected successfully.

Highlighted

Re: Not able to retrieve metrics that I just posted on HDP 2.6 sandbox

Rising Star

Thanks Bryan, You are right about the connection string. Anyway, following @Jay SenSharma suggestion, I used phoenix provided with HDP (in /usr/hdp/2.6.0.3-8/phoenix/ folder) and successfully connected to embedded HBase, no need to download an older version of phoenix.

Don't have an account?
Coming from Hortonworks? Activate your account here