Heartbeats can be lost if an exception occurs while Ambari Server is handling the heartbeat. It can also happen if there is an SSL certificate issue between server and agent. Can you please attach the ambari-server log and a log from the ambari-agent? ... View more

To help you, we'd need some more information: - Which version of HDP are you actually running on currently? Is it 2.5.3.0-37? - Can you post the entire output from the install command? - What is the content of /usr/hdp on this host which is having trouble? ... View more

Yes, my example is correct. There is no way to query directly for a specific property; you can only query by bean name. However, for alerts, we use a slash as a delimiter. The metric alert will remove the "VolumeFailuresTotal" and retrieve the "Hadoop:service=NameNode,name=FSNamesystemState" bean. Then it will extract the "VolumeFailuresTotal" metric. ... View more

Sure, you'd need to execute a POST to create the new alert: POST api/v1/clusters/<cluster-name>/alert_definitions { "AlertDefinition": { "component_name": "NAMENODE", "description": "This service-level alert is triggered if the total number of volume failures across the cluster is greater than the configured critical threshold.", "enabled": true, "help_url": null, "ignore_host": false, "interval": 2, "label": "NameNode Volume Failures", "name": "namenode_volume_failures", "scope": "ANY", "service_name": "HDFS", "source": { "jmx": { "property_list": [ "Hadoop:service=NameNode,name=FSNamesystemState/VolumeFailuresTotal" ], "value": "{0}" }, "reporting": { "ok": { "text": "There are {0} volume failures" }, "warning": { "text": "There are {0} volume failures", "value": 1 }, "critical": { "text": "There are {0} volume failures", "value": 1 }, "units": "Volume(s)" }, "type": "METRIC", "uri": { "http": "{{hdfs-site/dfs.namenode.http-address}}", "https": "{{hdfs-site/dfs.namenode.https-address}}", "https_property": "{{hdfs-site/dfs.http.policy}}", "https_property_value": "HTTPS_ONLY", "kerberos_keytab": "{{hdfs-site/dfs.web.authentication.kerberos.keytab}}", "kerberos_principal": "{{hdfs-site/dfs.web.authentication.kerberos.principal}}", "default_port": 0, "connection_timeout": 5, "high_availability": { "nameservice": "{{hdfs-site/dfs.internal.nameservices}}", "alias_key": "{{hdfs-site/dfs.ha.namenodes.{{ha-nameservice}}}}", "http_pattern": "{{hdfs-site/dfs.namenode.http-address.{{ha-nameservice}}.{{alias}}}}", "https_pattern": "{{hdfs-site/dfs.namenode.https-address.{{ha-nameservice}}.{{alias}}}}" } } } } } This will create a new METRIC alert which runs every 2 minutes. ... View more

Currently no, there is not. I believe there is a Jira open for changing how we send arguments to a script dispatcher so they are parameterized. Some fields, such as host name and component name, are not always present, so the current model simply omits them. ... View more

I think this goes back to the whole "dead is bad" theory. If I recall correctly, there was a metric Ambari was monitoring once on HBase - it was for "Dead RegionServers". We incorrectly assumed that "dead" was "bad". Because of this, while decommissioning a RegionServer, alerts would trigger (and not go away for a long time). In the end, it was determined that this metric wasn't really something which needed alerting on. HDFS is a little different - I believe that a DataNode is marked as stale if it hasn't reported in within 30 seconds and marked as dead if it hasn't reported within 1 minute. The problem here is that action is taken by the NameNode in this case - it will begin replicating blocks when it believes a DataNode is dead. So, we alert on it since it's something that is actively causing changes in the cluster data. The NameNode actually has metrics for differentiating "dead" vs "decommissiong dead": "NumLiveDataNodes": 3, "NumDeadDataNodes": 1, "NumDecomLiveDataNodes": 0, "NumDecomDeadDataNodes": 1, In the above example, Ambari won't worry about dead nodes which are marked as known decommissioning, but we will worry about this which are unexpected. ... View more