Can not get cluster ambari metrics

[#1]

Dear community,

I am trying to get cluster metrics on hdp 2.6 on centos 7, but when executing this example:

http://<ambari-server>:8080/api/v1/clusters/<cluster-name>?fields=metrics/load

I recieve empty response.

curl output:

HTTP/1.1 200 OK
Server: openresty/1.11.2.4
Date: Fri, 04 Aug 2017 04:49:50 GMT
Content-Type: text/plain
Content-Length: 155
Connection: keep-alive
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Cache-Control: no-store
Pragma: no-cache
Set-Cookie: AMBARISESSIONID=1p21o8xq7zkhq1o695y5wqcpgp;Path=/;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
User: admin
Vary: Accept-Encoding, User-Agent

Best regards,

Vladislav

[#2]

@Vladislav Falfushinsky

Your Metrics URL is not correct (It is missing the 1-min, CPUs, Nodes, Procs and the aggregator name).

Can you please try any of the following format.

http://localhost:8080/api/v1/clusters/Sandbox/?fields=metrics/load/1-min._avg[1501902776,1501906376,15]

http://localhost:8080/api/v1/clusters/Sandbox/?fields=metrics/load/CPUs._avg[1501902776,1501906376,15]

http://localhost:8080/api/v1/clusters/Sandbox/?fields=metrics/load/Nodes._avg[1501902776,1501906376,15],

http://localhost:8080/api/v1/clusters/Sandbox/?fields=metrics/load/Procs._avg[1501902776,1501906376,15]

Here "1501902776" and "1501906376" are the EPOC time. You can get the current (of some older EPOC) time using some online sites like http://www.epochconverter.com/ OR use the following command to get the current epoc time.

# date +%s
1501906757

.

[#3]

Dear @Jay SenSharma

Thanks for fast reply.

I`ve tried to run queries against different clusters, but receive strange errors:

curl -k --user admin:admin -H 'X-Requested-By:ambari' -X GET "https://mycluster:8443/api/v1/clusters/dev_sap/?fields=metrcs/load/CPUs._avg[1501902776,1501906376,15]"
curl: (3) [globbing] bad range in column 101
curl -k --user admin:admin -H 'X-Requested-By:ambari' -X GET "https://mycluster:8443/api/v1/clusters/dev_sap/?fields=metrics/load/Nodes._avg[1501902776,1501906376,15]"
curl: (3) [globbing] bad range in column 103
curl -k --user admin:admin -H 'X-Requested-By:ambari' -X GET "https://mycluster:8443/api/v1/clusters/dev_sap/?fields=metrics/load/1-min._avg[1501902776,1501906376,15]"
curl: (3) [globbing] bad range in column 103

[#4]

@Vladislav Falfushinsky

While using Curl you should use the Escape Characters for [ and ] symbols. Like Following:

curl -k --user admin:admin -H 'X-Requested-By:ambari' -X GET "https://mycluster:8443/api/v1/clusters/dev_sap/?fields=metrics/load/CPUs._avg\[1501936589,1501940189,15\]"

.

Please notice the escaped value:

CPUs._avg\[1501936589,1501940189,15\]

.

In my case i tried it on my sandbox and i got the results :

# # curl -k --user admin:admin -H 'X-Requested-By:ambari' -X GET  "http://localhost:8080/api/v1/clusters/Sandbox/?fields=metrics/load/CPUs._avg\[1501936589,1501940189,15\]"
{
  "href" : "http://localhost:8080/api/v1/clusters/Sandbox/?fields=metrics/load/CPUs._avg[1501936589,1501940189,15]",
  "Clusters" : {
    "cluster_name" : "Sandbox",
    "version" : "HDP-2.6"
  },
  "metrics" : {
    "load" : {
      "CPUs._avg" : [
        [
          0.0,
          1501936589
        ],
        [
          0.0,
          1501936619
        ],
        [
          0.0,
          1501936649
        ],
        [
          0.0,
          1501936679
        ],
        [
          0.0,
          1501936709
        ],
        [
          0.0,
          1501936739
        ],
        [
          0.0,
          1501936769
        ],
        [
          0.0,
          1501936799
        ],
        [
          0.0,
          1501936829
        ],
        [
          0.0,
          1501936859
        ],
        [
          0.0,
          1501936889
        ],
        [
          0.0,
          1501936919
        ],
        [
          0.0,
          1501936949
        ],
        [
          0.0,
          1501936979
        ],
        [
          0.0,
          1501937009
        ],
        [
          0.0,
          1501937039
        ],
        [
          0.0,
          1501937069
        ],
        [
          0.0,
          1501937099
        ],
        [
          0.0,
          1501937129
        ],
        [
          0.0,
          1501937159
        ],
        [
          0.0,
          1501937189
        ],
        [
          0.0,
          1501937219
        ],
        [
          0.0,
          1501937249
        ],
        [
          0.0,
          1501937279
        ],
        [
          0.0,
          1501937309
        ],
        [
          0.0,
          1501937339
        ],
        [
          0.0,
          1501937369
        ],
        [
          0.0,
          1501937399
        ],
        [
          0.0,
          1501937429
        ],
        [
          0.0,
          1501937459
        ],
        [
          0.0,
          1501937489
        ],
        [
          0.0,
          1501937519
        ],
        [
          0.0,
          1501937549
        ],
        [
          0.0,
          1501937579
        ],
        [
          0.0,
          1501937609
        ],
        [
          0.0,
          1501937639
        ],
        [
          0.0,
          1501937669
        ],
        [
          0.0,
          1501937699
        ],
        [
          0.0,
          1501937729
        ],
        [
          0.0,
          1501937759
        ],
        [
          0.0,
          1501937789
        ],
        [
          0.0,
          1501937819
        ],
        [
          0.0,
          1501937849
        ],
        [
          0.0,
          1501937879
        ],
        [
          0.0,
          1501937909
        ],
        [
          0.0,
          1501937939
        ],
        [
          0.0,
          1501937969
        ],
        [
          0.0,
          1501937999
        ],
        [
          0.0,
          1501938029
        ],
        [
          0.0,
          1501938059
        ],
        [
          0.0,
          1501938089
        ],
        [
          0.0,
          1501938119
        ],
        [
          0.0,
          1501938149
        ],
        [
          0.0,
          1501938179
        ],
        [
          0.0,
          1501938209
        ],
        [
          0.0,
          1501938239
        ],
        [
          0.0,
          1501938269
        ],
        [
          0.0,
          1501938299
        ],
        [
          0.0,
          1501938329
        ],
        [
          0.0,
          1501938359
        ],
        [
          0.0,
          1501938389
        ],
        [
          0.0,
          1501938419
        ],
        [
          0.0,
          1501938449
        ],
        [
          0.0,
          1501938479
        ],
        [
          0.0,
          1501938509
        ],
        [
          0.0,
          1501938539
        ],
        [
          0.0,
          1501938569
        ],
        [
          0.0,
          1501938599
        ],
        [
          0.0,
          1501938629
        ],
        [
          0.0,
          1501938659
        ],
        [
          0.0,
          1501938689
        ],
        [
          0.0,
          1501938719
        ],
        [
          0.0,
          1501938749
        ],
        [
          0.0,
          1501938779
        ],
        [
          0.0,
          1501938809
        ],
        [
          0.0,
          1501938839
        ],
        [
          0.0,
          1501938869
        ],
        [
          0.0,
          1501938899
        ],
        [
          0.0,
          1501938929
        ],
        [
          0.0,
          1501938959
        ],
        [
          0.0,
          1501938989
        ],
        [
          0.0,
          1501939019
        ],
        [
          0.0,
          1501939049
        ],
        [
          0.0,
          1501939079
        ],
        [
          0.0,
          1501939109
        ],
        [
          0.0,
          1501939139
        ],
        [
          0.0,
          1501939169
        ],
        [
          0.0,
          1501939199
        ],
        [
          0.0,
          1501939229
        ],
        [
          0.0,
          1501939259
        ],
        [
          0.0,
          1501939289
        ],
        [
          0.0,
          1501939319
        ],
        [
          0.0,
          1501939349
        ],
        [
          0.0,
          1501939379
        ],
        [
          0.0,
          1501939409
        ],
        [
          0.0,
          1501939439
        ],
        [
          0.0,
          1501939469
        ],
        [
          0.0,
          1501939499
        ],
        [
          0.0,
          1501939529
        ],
        [
          0.0,
          1501939559
        ],
        [
          0.0,
          1501939589
        ],
        [
          0.0,
          1501939619
        ],
        [
          0.0,
          1501939649
        ],
        [
          0.0,
          1501939679
        ],
        [
          0.0,
          1501939709
        ],
        [
          0.0,
          1501939739
        ],
        [
          0.0,
          1501939769
        ],
        [
          0.0,
          1501939799
        ],
        [
          0.0,
          1501939829
        ],
        [
          0.0,
          1501939859
        ],
        [
          4.0,
          1501939860
        ],
        [
          4.0,
          1501939890
        ],
        [
          4.0,
          1501939920
        ],
        [
          4.0,
          1501939950
        ],
        [
          4.0,
          1501939980
        ],
        [
          4.0,
          1501940010
        ],
        [
          4.0,
          1501940040
        ],
        [
          4.0,
          1501940070
        ],
        [
          0.0,
          1501940100
        ],
        [
          0.0,
          1501940130
        ],
        [
          0.0,
          1501940160
        ]
      ]
    }
  }
}

.

[#5]

@Jay SenSharma

Escaping characters helped, but still receive an empty response:

curl --verbose --user admin:admin -H 'X-Requested-By:ambari' -H 'X-Forwarded-To:foo-controller1.local:31080' -X GET "http://192.168.1.14/api/v1/clusters/foo/?fields=metrics/load/CPUs\[1501965010,1501965087,15\]"
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 192.168.1.14...
* Connected to 192.168.1.14 (192.168.1.14) port 80 (#0)
* Server auth using Basic with user 'admin'
> GET /api/v1/clusters/foo/?fields=metrics/load/CPUs[1501965010,1501965087,15] HTTP/1.1
> Host: 192.168.1.14
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.47.0
> Accept: */*
> X-Requested-By:ambari
> X-Forwarded-To:foo-controller1.local:31080
>
< HTTP/1.1 200 OK
< Server: openresty/1.11.2.4
< Date: Fri, 04 Aug 2017 10:42:10 GMT
< Content-Type: text/plain
< Content-Length: 184
< Connection: keep-alive
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Cache-Control: no-store
< Pragma: no-cache
< Set-Cookie: AMBARISESSIONID=ao8jm5334pchda97v8yafnhx;Path=/;HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< User: admin
< Vary: Accept-Encoding, User-Agent
<
{
  "href" : "http://192.168.1.14/api/v1/clusters/foo/?fields=metrics/load/CPUs[1501965010,1501965087,15]",
  "Clusters" : {
    "cluster_name" : "foo",
    "version" : "HDP-2.6"
  }
* Connection #0 to host 192.168.1.14 left intact

Also tried, but the result is the same:

curl --verbose --user admin:admin -H 'X-Requested-By:ambari' -H 'X-Forwarded-To:foo-controller1.local:31080' -X GET "http://192.168.1.14/api/v1/clusters/foo/?fields=metrics/cpu/Idle\[1501965010,1501965087,15\]"

[#6]

@Vladislav Falfushinsky

Good to know that at least now it is returning a json response.

The Data is not coming, that indicates your Metrics Monitors might not be running on the individual hosts. So please check if you have the Metrics Monitors Running fine or not? Please check the "/var/log/ambari-metris-monitor/ambari-metrics-monitor.out" file to see if there are any errors?

Now the problem is outside of API Access, Because empty response indicates that the CPU load data is not at all present in the AMS collector Database, So in ambari UI as well you will not be able to see the CPU load data in the Ambari Dashboard.

We will need to see what is the issue with the Metrics Monitors.

[#7]

Thanks for help. Finally got some data. There were several problems:

1) Sometimes metrics collector had been failing.

2) Time had not been synchronized. Host which requested metrics had different time rather then ambari cluster.