Support Questions
Find answers, ask questions, and share your expertise

ambari alert NameNode Client RPC Processing Latency status UNKNOW

Expert Contributor

I upgrade ambari 2.2 to 2.5.1 recently.

after upgrade,I found that,some alert status is UNKNOW.

open an alert definition, log shows :

Properties file doesn't contain namenode.sink.timeline.collector.hosts. Can't define metric collector hosts

but when I open the service tap and the metrics shows the data:

So I don't why.

And I view the configure file:/etc/hadoop/2.4.0.0-169/0/hadoop-metrics2.properties

it show :

datanode.sink.timeline.collector=node4.hadoop
namenode.sink.timeline.collector=node4.hadoop
resourcemanager.sink.timeline.collector=node4.hadoop
nodemanager.sink.timeline.collector=node4.hadoop
jobhistoryserver.sink.timeline.collector=node4.hadoop
journalnode.sink.timeline.collector=node4.hadoop
applicationhistoryserver.sink.timeline.collector=node4.hadoop

40615-ambari-alert-unknow-metrics.png

40614-ambari-alert-unknow-response.png

40613-ambari-alert-unknow.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

Expert Contributor

@Aravindan Vijayan

I change the file '/var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2' ,and restart ambari server and agent ,but there is no use.

But I change the content of /etc/hadoop/conf/hadoop-metrics2.properties and restart ambari server and agent ,it goes ok.

View solution in original post

5 REPLIES 5

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

Super Mentor

@darkz yu
The Error indicates that your "/etc/hadoop/conf/hadoop-metrics2.properties" file does not have the property with exact name as "namenode.sink.timeline.collector.hosts"


The Example file should look like following:

# grep 'sink.timeline.collector.hosts'  /etc/hadoop/conf/hadoop-metrics2.properties
datanode.sink.timeline.collector.hosts=amb25102.example.com
resourcemanager.sink.timeline.collector.hosts=amb25102.example.com
nodemanager.sink.timeline.collector.hosts=amb25102.example.com
jobhistoryserver.sink.timeline.collector.hosts=amb25102.example.com
journalnode.sink.timeline.collector.hosts=amb25102.example.com
maptask.sink.timeline.collector.hosts=amb25102.example.com
reducetask.sink.timeline.collector.hosts=amb25102.example.com
applicationhistoryserver.sink.timeline.collector.hosts=amb25102.example.com




Notice there is "nodemanager.sink.timeline.collector.hosts" (hosts) string added to those properties.


If you have upgraded ambari from 2.2 version to ambari 2.5.1 then please double check if you have followed the Ambari Post Upgrade Steps. The Ambari Post upgrade steps involved Upgrading the Ambari Metrics Services as well like: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-upgrade/content/upgrade_ambari_me...


Other Post upgrade Steps are mentioned here : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-upgrade/content/post_ambari_upgra...


Can you please share the output of the following command from AMS collector host to see if the AMS binaries were properly upgraded or not?

#  rpm -qa | grep ambari
ambari-server-2.5.1.0-159.x86_64
ambari-metrics-monitor-2.5.1.0-159.x86_64
ambari-agent-2.5.1.0-159.x86_64
ambari-metrics-hadoop-sink-2.5.1.0-159.x86_64
ambari-metrics-grafana-2.5.1.0-159.x86_64

.

.

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

Expert Contributor

@darkz yu

This is a known regression fixed in Ambari 2.5.2 (https://issues.apache.org/jira/browse/AMBARI-21328)

To workaround, you can

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

Expert Contributor
@Aravindan Vijayan

Let me try.

And however,there are some table not found error:

Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ambari.ds_jobimpl_6' doesn't exist

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

Expert Contributor

@Aravindan Vijayan

I change the file '/var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2' ,and restart ambari server and agent ,but there is no use.

But I change the content of /etc/hadoop/conf/hadoop-metrics2.properties and restart ambari server and agent ,it goes ok.

View solution in original post

Re: ambari alert NameNode Client RPC Processing Latency status UNKNOW

Expert Contributor
@darkz yu

Yes, that sounds right. Sorry I missed the step which would have triggered the modification of the '/etc/hadoop/conf/hadoop-metrics2.properties' file. So, is your problem fixed now?