question Re: Check Ambari Metrics + service check in Archives of Support Questions (Read Only)

Check Ambari Metrics + service check

mike_bronson7 — Sun, 18 Aug 2019 03:15:40 GMT

we perform service check to Ambari Metrics

and we get the following errors - "All metrics collectors are unavailable"

what we can do regarding that , in order to solve the problem?

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py", line 207, in <module>
    AMSServiceCheck().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py", line 159, in service_check
    raise Fail("All metrics collectors are unavailable.")
resource_management.core.exceptions.Fail: All metrics collectors are unavailable.

Re: Check Ambari Metrics + service check

jsensharma — Wed, 11 Apr 2018 16:01:07 GMT

@Michael Bronson

Few checks will be good to perform:

1. To verify is ambari is showing false Running AMS services? Please check the AMS collector hosts if the AMS collector is actually running and listening to the correct address/port:

# netstat -tnlpa | grep 6188
# hostname -f

2. Please verify if the AMS process PID are matching with the PIDs that are listed int he following files. Some times not matching PIDs causes false info in the UI

$  ps -ef | grep ^ams | grep ApplicationHistoryServer

$  cat /var/run/ambari-metrics-collector/ambari-metrics-collector.pid  

$  ps -ef | grep ^ams | grep HMaster

$  cat /var/run/ambari-metrics-collector/hbase-ams-master.pid

3. Can you try restarting the AMS collector Service once to see if you notice any error int he AMS collector logs?

Re: Check Ambari Metrics + service check

jsensharma — Wed, 11 Apr 2018 16:04:14 GMT

@Michael Bronson

Also please share the output of the following command from AMS collector host and few of the cluster nodes to verify if the AMS service version is same as Ambari Binary version or not?

# rpm -qa | grep ambari

Re: Check Ambari Metrics + service check

mike_bronson7 — Wed, 11 Apr 2018 16:11:44 GMT

rpm -qa | grep ambari
ambari-metrics-monitor-2.6.1.0-143.x86_64
ambari-metrics-hadoop-sink-2.6.1.0-143.x86_64
ambari-agent-2.6.1.0-143.x86_64

Re: Check Ambari Metrics + service check

mike_bronson7 — Wed, 11 Apr 2018 16:13:57 GMT

abouit "restarting the AMS" , when we restart all metrics service it will also restart the AMS?

Re: Check Ambari Metrics + service check

mike_bronson7 — Wed, 11 Apr 2018 16:17:32 GMT

@Jay after we set the corect date on all machines , now ambari metrics service check is ok , do you think this is logical ?

Re: Check Ambari Metrics + service check

jsensharma — Wed, 11 Apr 2018 16:55:15 GMT

@Michael Bronson

Yes, the AMS basically posts a dummy metrics to the collector with start & End time as mentioned int eh script (which is a relative time) so time difference can be a valid reason for failing service checks.

      get_metrics_parameters = {
        "metricNames": "AMBARI_METRICS.SmokeTest.FakeMetric",
        "appId": "amssmoketestfake",
        "hostname": params.hostname,
        "startTime": current_time - 60000,
        "endTime": current_time + 61000,
        "precision": "seconds",
        "grouped": "false",
      }

https://github.com/apache/ambari/blob/release-2.6.1/ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py#L83-L84

And

https://github.com/apache/ambari/blob/release-2.6.1/ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/service_check.py#L121-L123

Re: Check Ambari Metrics + service check

jsensharma — Wed, 11 Apr 2018 17:05:11 GMT

@Michael Bronson

Yes, Ambari UI --> Ambari Metrics --> Service Actions --> Restart All Metrics service will cause Ambari Metrics Monitor/ Grafana & Metrics Collector restart.