Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ambari alerts strange

ambari alerts strange

Super Collaborator

Hi:

In one node i have this alert, but i dont know what is this:

WARNING 2016-07-12 13:10:33,654 base_alert.py:140 - [Alert][datanode_storage] Unable to execute alert. [Alert][datanode_storage] Unable to extract JSON from JMX respons
11 REPLIES 11

Re: ambari alerts strange

Super Collaborator

Is this cluster Kerberized? If so, there's a known issue in some versions of Ambari where a Kerberos ticket invalidation causes the JSON JMX data not to be able to be retrieved. My guess is that this issue is intermittent and that your are running with Kerberos.

If my assumptions are wrong and this is a persistent alert, can you check to see if you can get data from the JMX endpoint of the DataNode? http://<datanode>:50075/jmx

Re: ambari alerts strange

Super Collaborator

yes i can, maybe the alert is wrong, how can i remove the alert from ambari database??

{
  "beans" : [ {
    "name" : "JMImplementation:type=MBeanServerDelegate",
    "modelerType" : "javax.management.MBeanServerDelegate",
    "MBeanServerId" : "a01hop07_1468312335324",
    "SpecificationName" : "Java Management Extensions",
    "SpecificationVersion" : "1.4",
    "SpecificationVendor" : "Oracle Corporation",
    "ImplementationName" : "JMX",
    "ImplementationVersion" : "1.8.0_60-b27",
    "ImplementationVendor" : "Oracle Corporation"
  }, {
    "name" : "java.lang:type=Runtime",
    "modelerType" : "sun.management.RuntimeImpl",
    "SpecName" : "Java Virtual Machine Specification",
    "SpecVendor" : "Oracle Corporation",
    "SpecVersion" : "1.8",
    "ManagementSpecVersion" : "1.2",

Re: ambari alerts strange

Super Collaborator

OK, so that's good. What about my original comment about the cluster being Kerberized. If it is, you are mostly likely hitting https://issues.apache.org/jira/browse/AMBARI-15324.

Re: ambari alerts strange

@Roberto Sancho

It indicates whether DataNode is down, (OR) Due to some N/W issue (OR) long Garbage Collection pause the Datanode was not accessible. This message comes from "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py" script and following is the logic:

if isinstance(self.metric_info, JmxMetric):
  jmx_property_values, http_code = self._load_jmx(alert_uri.is_ssl_enabled, host, port, self.metric_info)
  if not jmx_property_values and http_code in [200, 307]:
    collect_result = self.RESULT_UNKNOWN
    value_list.append('HTTP {0} response (metrics unavailable)'.format(str(http_code)))
  elif not jmx_property_values and http_code not in [200, 307]:
    raise Exception("[Alert][{0}] Unable to extract JSON from JMX response".format(self.get_name()))

Re: ambari alerts strange

@Roberto Sancho

Directly editing DB is not recommended. However if you are using Prior version of ambari 2.2 then you can get rid of those alerts by adding 'misfire_grace_time':10 to APS_CONFIG in /usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py on every node. After the update that section should read:

APS_CONFIG = { 
'threadpool.core_threads': 3, 
'coalesce': True, 
'standalone': False, 
'misfire_grace_time':10 
}

In this we are allowing up to 10 seconds for all tests to complete. After that restart all ambari_agents. We tried on one cluster and it worked. This is most likely fixed in ambari-2.2 but happens in 2.1.2.

Which ambari version are you using? Because from Ambari 2.2.x, to resolve this stale alert issue, you may increase alert_grace_period to 15 in /etc/ambari-agent/conf/ambari-agent.ini and start agent.

Try disabling and then after 5-10 seconds later enabling that alert definition from ambari to see if that Stale Alert Goes away.

Re: ambari alerts strange

Super Collaborator

The alert isn't misfiring; it's running, but hitting an error, so this won't resolve the issue.

Highlighted

Re: ambari alerts strange

New Contributor

Can the metrics refresh self? I tried disabling and enabling and it indeed worked. But why we should manually refresh it?

Re: ambari alerts strange

@Roberto Sancho

If the stale alert does not go away even after making above changes then as a last option you can try deleting that particular alert record from the "alert_current" table inside your "ambari" DB.

But before making any direct change to the ambari DB it is always recommended to take a DB back.

Re: ambari alerts strange

Super Collaborator

It's not a stale alert, so that won't help.

Don't have an account?
Coming from Hortonworks? Activate your account here