Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ambari service check failed on metrics service

ambari service check failed on metrics service

hi all



we run the service check on metrics collector , this is part of verification before we upgrade the HDP to 2.6.5

please advice what is the reason for this failed


the logs are


Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py", line 207, in <module>
    AMSServiceCheck().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py", line 159, in service_check
    raise Fail("All metrics collectors are unavailable.")
resource_management.core.exceptions.Fail: All metrics collectors are unavailable.


2019-07-16 19:43:26,585 - Using hadoop conf dir: /usr/hdp/2.6.4.0-91/hadoop/conf
2019-07-16 19:43:26,595 - checked_call['hostid'] {}
2019-07-16 19:43:26,607 - checked_call returned (0, 'a40a4d5c')
2019-07-16 19:43:26,608 - Ambari Metrics service check was started.
2019-07-16 19:43:26,645 - Generated metrics for host master02.sys72.com :
{
  "metrics": [
    {
      "metricname": "AMBARI_METRICS.SmokeTest.FakeMetric",
      "appid": "amssmoketestfake",
      "hostname": "worker01.sys72.com",
      "timestamp": 1563306206000,
      "starttime": 1563306206000,
      "metrics": {
        "1563306206000": 0.209009290753,
        "1563306207000": 1563306206000
      }
    }
  ]
}
2019-07-16 19:43:26,646 - Connecting (POST) to master02.sys72.com:6188/ws/v1/timeline/metrics/
2019-07-16 19:43:26,676 - Http response for host master02.sys72.com: 200 OK
2019-07-16 19:43:26,677 - Http data: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><response/>
2019-07-16 19:43:26,677 - Metrics were saved.
2019-07-16 19:43:26,678 - Connecting (GET) to master02.sys72.com:6188/ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric&hostname=worker01.sys72.com&precision=seconds&grouped=false&startTime=1563306146000&appId=amssmoketestfake&endTime=1563306267000
2019-07-16 19:43:26,723 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:43:26,724 - Http data: {"metrics":[]}
2019-07-16 19:43:26,724 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:43:26,725 - Values weren't stored yet. Retrying in 10 seconds.
2019-07-16 19:43:36,744 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:43:36,744 - Http data: {"metrics":[]}
2019-07-16 19:43:36,745 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:43:36,745 - Values weren't stored yet. Retrying in 10 seconds.
2019-07-16 19:43:46,766 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:43:46,767 - Http data: {"metrics":[]}
2019-07-16 19:43:46,767 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:43:46,767 - Values weren't stored yet. Retrying in 10 seconds.
2019-07-16 19:43:56,796 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:43:56,797 - Http data: {"metrics":[]}
2019-07-16 19:43:56,797 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:43:56,797 - Values weren't stored yet. Retrying in 10 seconds.
2019-07-16 19:44:06,824 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:44:06,825 - Http data: {"metrics":[]}
2019-07-16 19:44:06,825 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:44:06,825 - Ambari Metrics service check failed on collector host master02.sys72.com. Reason : Values 0.209009290753 and 1563306206000 were not found in the response.
2019-07-16 19:44:06,829 - Exception while running function '<bound method AMSServiceCheck.service_check_for_single_host of <__main__.AMSServiceCheck object at 0x28cedd0>>' for 'master02.sys72.com'. Reason : Ambari Metrics service check failed on collector host master02.sys72.com. Reason : Values 0.209009290753 and 1563306206000 were not found in the response.



Michael-Bronson
Don't have an account?
Coming from Hortonworks? Activate your account here