hi all we run the service check on metrics collector , this is part of verification before we upgrade the HDP to 2.6.5 please advice what is the reason for this failed the logs are Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py", line 207, in <module>
AMSServiceCheck().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
method(env)
File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py", line 159, in service_check
raise Fail("All metrics collectors are unavailable.")
resource_management.core.exceptions.Fail: All metrics collectors are unavailable.
2019-07-16 19:43:26,585 - Using hadoop conf dir: /usr/hdp/2.6.4.0-91/hadoop/conf
2019-07-16 19:43:26,595 - checked_call['hostid'] {}
2019-07-16 19:43:26,607 - checked_call returned (0, 'a40a4d5c')
2019-07-16 19:43:26,608 - Ambari Metrics service check was started.
2019-07-16 19:43:26,645 - Generated metrics for host master02.sys72.com :
{
"metrics": [
{
"metricname": "AMBARI_METRICS.SmokeTest.FakeMetric",
"appid": "amssmoketestfake",
"hostname": "worker01.sys72.com",
"timestamp": 1563306206000,
"starttime": 1563306206000,
"metrics": {
"1563306206000": 0.209009290753,
"1563306207000": 1563306206000
}
}
]
}
2019-07-16 19:43:26,646 - Connecting (POST) to master02.sys72.com:6188/ws/v1/timeline/metrics/
2019-07-16 19:43:26,676 - Http response for host master02.sys72.com: 200 OK
2019-07-16 19:43:26,677 - Http data: <?xml version="1.0" encoding="UTF-8" standalone="yes"?><response/>
2019-07-16 19:43:26,677 - Metrics were saved.
2019-07-16 19:43:26,678 - Connecting (GET) to master02.sys72.com:6188/ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric&hostname=worker01.sys72.com&precision=seconds&grouped=false&startTime=1563306146000&appId=amssmoketestfake&endTime=1563306267000
2019-07-16 19:43:26,723 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:43:26,724 - Http data: {"metrics":[]}
2019-07-16 19:43:26,724 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:43:26,725 - Values weren't stored yet. Retrying in 10 seconds.
2019-07-16 19:43:36,744 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:43:36,744 - Http data: {"metrics":[]}
2019-07-16 19:43:36,745 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:43:36,745 - Values weren't stored yet. Retrying in 10 seconds.
2019-07-16 19:43:46,766 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:43:46,767 - Http data: {"metrics":[]}
2019-07-16 19:43:46,767 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:43:46,767 - Values weren't stored yet. Retrying in 10 seconds.
2019-07-16 19:43:56,796 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:43:56,797 - Http data: {"metrics":[]}
2019-07-16 19:43:56,797 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:43:56,797 - Values weren't stored yet. Retrying in 10 seconds.
2019-07-16 19:44:06,824 - Http response for host master02.sys72.com : 200 OK
2019-07-16 19:44:06,825 - Http data: {"metrics":[]}
2019-07-16 19:44:06,825 - Metrics were retrieved from host master02.sys72.com
2019-07-16 19:44:06,825 - Ambari Metrics service check failed on collector host master02.sys72.com. Reason : Values 0.209009290753 and 1563306206000 were not found in the response.
2019-07-16 19:44:06,829 - Exception while running function '<bound method AMSServiceCheck.service_check_for_single_host of <__main__.AMSServiceCheck object at 0x28cedd0>>' for 'master02.sys72.com'. Reason : Ambari Metrics service check failed on collector host master02.sys72.com. Reason : Values 0.209009290753 and 1563306206000 were not found in the response.
... View more
