ambari alert NameNode Client RPC Processing Latency status UNKNOW

[#1]

I upgrade ambari 2.2 to 2.5.1 recently.

after upgrade,I found that,some alert status is UNKNOW.

open an alert definition, log shows :

Properties file doesn't contain namenode.sink.timeline.collector.hosts. Can't define metric collector hosts

but when I open the service tap and the metrics shows the data:

So I don't why.

And I view the configure file:/etc/hadoop/2.4.0.0-169/0/hadoop-metrics2.properties

it show :

datanode.sink.timeline.collector=node4.hadoop
namenode.sink.timeline.collector=node4.hadoop
resourcemanager.sink.timeline.collector=node4.hadoop
nodemanager.sink.timeline.collector=node4.hadoop
jobhistoryserver.sink.timeline.collector=node4.hadoop
journalnode.sink.timeline.collector=node4.hadoop
applicationhistoryserver.sink.timeline.collector=node4.hadoop

[#2]

@darkz yu
The Error indicates that your "/etc/hadoop/conf/hadoop-metrics2.properties" file does not have the property with exact name as "namenode.sink.timeline.collector.hosts"

The Example file should look like following:

# grep 'sink.timeline.collector.hosts'  /etc/hadoop/conf/hadoop-metrics2.properties
datanode.sink.timeline.collector.hosts=amb25102.example.com
resourcemanager.sink.timeline.collector.hosts=amb25102.example.com
nodemanager.sink.timeline.collector.hosts=amb25102.example.com
jobhistoryserver.sink.timeline.collector.hosts=amb25102.example.com
journalnode.sink.timeline.collector.hosts=amb25102.example.com
maptask.sink.timeline.collector.hosts=amb25102.example.com
reducetask.sink.timeline.collector.hosts=amb25102.example.com
applicationhistoryserver.sink.timeline.collector.hosts=amb25102.example.com

Notice there is "nodemanager.sink.timeline.collector.hosts" (hosts) string added to those properties.

If you have upgraded ambari from 2.2 version to ambari 2.5.1 then please double check if you have followed the Ambari Post Upgrade Steps. The Ambari Post upgrade steps involved Upgrading the Ambari Metrics Services as well like: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-upgrade/content/upgrade_ambari_me...

Other Post upgrade Steps are mentioned here : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-upgrade/content/post_ambari_upgra...

Can you please share the output of the following command from AMS collector host to see if the AMS binaries were properly upgraded or not?

#  rpm -qa | grep ambari
ambari-server-2.5.1.0-159.x86_64
ambari-metrics-monitor-2.5.1.0-159.x86_64
ambari-agent-2.5.1.0-159.x86_64
ambari-metrics-hadoop-sink-2.5.1.0-159.x86_64
ambari-metrics-grafana-2.5.1.0-159.x86_64

.

.

[#3]

@darkz yu

This is a known regression fixed in Ambari 2.5.2 (https://issues.apache.org/jira/browse/AMBARI-21328)

To workaround, you can

• Replace the contents of the file '/var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2' with content from https://github.com/apache/ambari/blob/branch-2.5/ambari-server/src/main/resources/stacks/HDP/2.0.6/h...
• Restart Ambari Server.

[#4]

@Aravindan Vijayan

Let me try.

And however,there are some table not found error:

Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.ds_jobimpl_6' doesn't exist

[#5 · Accepted solution]

@Aravindan Vijayan

I change the file '/var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2' ,and restart ambari server and agent ,but there is no use.

But I change the content of /etc/hadoop/conf/hadoop-metrics2.properties and restart ambari server and agent ,it goes ok.

[#6]

@darkz yu

Yes, that sounds right. Sorry I missed the step which would have triggered the modification of the '/etc/hadoop/conf/hadoop-metrics2.properties' file. So, is your problem fixed now?