question Re: Ambari metrics in Archives of Support Questions (Read Only)

Ambari metrics

jimfrank324 — Fri, 16 Sep 2022 11:16:57 GMT

when i finish install the ambari , i view 'HDFS‘ in , in summary i can see “NameNode heap” data ，like this

but under "Metrics" no graphic data .like this

the same in "Dashboard", only memory usage,cpu usage,network

Re: Ambari metrics

jsensharma — Mon, 20 Mar 2017 09:49:39 GMT

@Elvis Zhang

Can you please check if the "metrics-monitor" component is running on all the hosts?

Please check:

# ps -ef | grep ^ams
ams      14933     1  0 02:50 ?        00:00:00 /usr/bin/python2.6 /usr/lib/python2.6/site-packages/resource_monitoring/main.py start

If it is not running then please start it and then check the Amabri UI again in few minutes.

Also please let us know which version of ambari is it? And do you see any error inside the following log?

# less /var/log//ambari-metrics-monitor/ambari-metrics-monitor.out    (all hosts)

Also please check if the following file is pointing to correct AMS collector host & port.

# less /etc/ambari-metrics-monitor/conf/metric_monitor.ini

Re: Ambari metrics

jimfrank324 — Sun, 18 Aug 2019 11:27:54 GMT

yes, i checked them , they were all installed on three hosts.

Re: Ambari metrics

jsensharma — Mon, 20 Mar 2017 10:04:59 GMT

@Elvis Zhang

In the following log file do you see any error?

/var/log/ambari-metrics-collector/ambari-metrics-collector.log

What is the ambari server ambari metrics collector version? Is it same on all hosts? Are you seeing the same version for all AMS & ambari components?

# rpm -qa | grep ambari*

Re: Ambari metrics

jsensharma — Mon, 20 Mar 2017 10:09:16 GMT

@Elvis Zhang

If you have just added AMS to your cluster then the services need to be restarted to initiate the Sink. That is controlled by metrics properties file like:

# cat /etc/hadoop/conf/hadoop-metrics2.properties

Also in the NameNode (DataNode) logs do you see that the "Sink timeline started" is started as following:

# less /var/log/hadoop/hdfs/hadoop-hdfs-namenode-erie1.example.com.log

INFO  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(82)) - Initializing Timeline metrics sink.
INFO  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(100)) - Identified hostname = erie1.example.com, serviceName = namenode
INFO  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(118)) - Collector Uri: <a href="http://erie2.example.com:6188/ws/v1/timeline/metrics">http://erie2.example.com:6188/ws/v1/timeline/metrics</a>
INFO  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(166)) - RPC port properties configured: {8020=client}
NFO  impl.MetricsSinkAdapter (MetricsSinkAdapter.java:start(206)) - Sink timeline started

Re: Ambari metrics

jimfrank324 — Mon, 20 Mar 2017 10:58:48 GMT

thanks Jay very much !! yes ,i find an error in ambari-metrics-monitor.out

[WARNING] emitter.py:84 - Error sending metrics to server. [Errno 113] No route to host

it seems that metrics-collector host can't be connect ,but i don't know how to config this ? does it ok when i installed by ambari-server?

here is my hosts in '/etc/hosts' (all hosts are same)

192.168.2.43   bn-001.hadoop 
192.168.2.39   bn-002.hadoop
192.168.2.38   bn-003.hadoop ( this is  ambari-metrics-collector)

here is “ metric_monitor.ini ” in my "metric collector host is "bn-001.hadoop"

[default]
debug_level = INFO
hostname = bn-001.hadoop
enable_time_threshold = false
enable_value_threshold = false 

[emitter]
send_interval = 60


[collector]
collector_sleep_interval = 10
max_queue_size = 5000
host = bn-003.hadoop
port = 6188
https_enabled = False

Re: Ambari metrics

jsensharma — Mon, 20 Mar 2017 12:04:49 GMT

@Elvis Zhang

It is strange to see the following error:

[WARNING] emitter.py:84 - Error sending metrics to server. [Errno 113] No route to host

Which means the ambari metrics monitors are not able to communicate to AMS collector on the mentioned port. hence the data (memory/CPU ...etc) are not being emitted. Can you please check if from the metrics monitor host you are able to connect to AMS collector host/port (Just to see that there is no Firewall restriction on the port.)

# telnet  bn-003.hadoop   6188

Also are the FQDNs correctly configured through out the cluster. So that you get the correct hostname in the following command output:

# hostname -f

Also please see: http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/set_the_hostname.html

and

http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/edit_the_network_configuration_file.html

Re: Ambari metrics

jimfrank324 — Sun, 18 Aug 2019 11:27:46 GMT

OK i fix it . My OS is 'Centos 7.1',

i alos used (Centos 6.*)

service iptables stop

So i change is with

systemctl stop firewalld

Every thing is ok Now , Thanks for Jay's help again.