question Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW in Archives of Support Questions (Read Only)

ambari alert NameNode Client RPC Processing Latency status UNKNOW

darkz1984 — Sun, 18 Aug 2019 05:07:40 GMT

I upgrade ambari 2.2 to 2.5.1 recently.

after upgrade,I found that,some alert status is UNKNOW.

open an alert definition, log shows :

Properties file doesn't contain namenode.sink.timeline.collector.hosts. Can't define metric collector hosts

but when I open the service tap and the metrics shows the data:

So I don't why.

And I view the configure file:/etc/hadoop/2.4.0.0-169/0/hadoop-metrics2.properties

it show :

datanode.sink.timeline.collector=node4.hadoop
namenode.sink.timeline.collector=node4.hadoop
resourcemanager.sink.timeline.collector=node4.hadoop
nodemanager.sink.timeline.collector=node4.hadoop
jobhistoryserver.sink.timeline.collector=node4.hadoop
journalnode.sink.timeline.collector=node4.hadoop
applicationhistoryserver.sink.timeline.collector=node4.hadoop

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

jsensharma — Tue, 03 Oct 2017 13:37:59 GMT

@darkz yu
The Error indicates that your "/etc/hadoop/conf/hadoop-metrics2.properties" file does not have the property with exact name as "namenode.sink.timeline.collector.hosts"

The Example file should look like following:

# grep 'sink.timeline.collector.hosts'  /etc/hadoop/conf/hadoop-metrics2.properties
datanode.sink.timeline.collector.hosts=amb25102.example.com
resourcemanager.sink.timeline.collector.hosts=amb25102.example.com
nodemanager.sink.timeline.collector.hosts=amb25102.example.com
jobhistoryserver.sink.timeline.collector.hosts=amb25102.example.com
journalnode.sink.timeline.collector.hosts=amb25102.example.com
maptask.sink.timeline.collector.hosts=amb25102.example.com
reducetask.sink.timeline.collector.hosts=amb25102.example.com
applicationhistoryserver.sink.timeline.collector.hosts=amb25102.example.com

Notice there is "nodemanager.sink.timeline.collector.hosts" (hosts) string added to those properties.

If you have upgraded ambari from 2.2 version to ambari 2.5.1 then please double check if you have followed the Ambari Post Upgrade Steps. The Ambari Post upgrade steps involved Upgrading the Ambari Metrics Services as well like: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-upgrade/content/upgrade_ambari_metrics.html

Other Post upgrade Steps are mentioned here : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-upgrade/content/post_ambari_upgrade_tasks.html

Can you please share the output of the following command from AMS collector host to see if the AMS binaries were properly upgraded or not?

#  rpm -qa | grep ambari
ambari-server-2.5.1.0-159.x86_64
ambari-metrics-monitor-2.5.1.0-159.x86_64
ambari-agent-2.5.1.0-159.x86_64
ambari-metrics-hadoop-sink-2.5.1.0-159.x86_64
ambari-metrics-grafana-2.5.1.0-159.x86_64

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

avijayan — Wed, 04 Oct 2017 01:53:23 GMT

@darkz yu

This is a known regression fixed in Ambari 2.5.2 (https://issues.apache.org/jira/browse/AMBARI-21328)

To workaround, you can

Replace the contents of the file '/var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2' with content from https://github.com/apache/ambari/blob/branch-2.5/ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2
Restart Ambari Server.

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

darkz1984 — Fri, 06 Oct 2017 11:06:59 GMT

@Aravindan Vijayan

Let me try.

And however,there are some table not found error:

Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.ds_jobimpl_6' doesn't exist

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

darkz1984 — Fri, 06 Oct 2017 11:29:04 GMT

@Aravindan Vijayan

I change the file '/var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2' ,and restart ambari server and agent ,but there is no use.

But I change the content of /etc/hadoop/conf/hadoop-metrics2.properties and restart ambari server and agent ,it goes ok.

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

avijayan — Thu, 12 Oct 2017 03:38:32 GMT

@darkz yu

Yes, that sounds right. Sorry I missed the step which would have triggered the modification of the '/etc/hadoop/conf/hadoop-metrics2.properties' file. So, is your problem fixed now?