Support Questions
Find answers, ask questions, and share your expertise

metrics monitor

Super Collaborator

hi:

I have the metrics running, but just i can see the informacion about one node, the last one, the are not alert, any suggestion???

7150-snip20160831-1.png

25 REPLIES 25

Super Collaborator

hi:

in this file

view /etc/ambari-metrics-monitor/conf/metric_monitor.ini

i can see that:

[default]
debug_level = INFO
metrics_server = xxxxxx07:6188
hostname = xxxxxxx06
enable_time_threshold = false
enable_value_threshold = false
[emitter]
send_interval = 60
[collector]
collector_sleep_interval = 5
max_queue_size = 5000

06 is the host where iam cheking the file and 07 is the server where metrics collector is running, so... i think everything look fine, any oher file where i need to see????

@Roberto Sancho

Have you tried restarting the ambari server and all of the ambari agents since you moved the AMS?

Also, have you taken a look at: https://cwiki.apache.org/confluence/display/AMBARI/Known+Issues

Super Collaborator

hi, y restarted agent and server now, but still not info

Contributor
@Roberto Sancho

Hi,

Can you check the datanode logs in /var/logs/hadoop/hdfs/hadoop-hdfs-datanode-*. If they are able to connect to the master nodes. May be can you upload your datanode logs from the other hosts here.

Super Collaborator

i checked the file on 06:

tail -200f  /var/log/hadoop/hdfs/hadoop-hdfs-datanode-xxxxxxx06.log

and tihs error

2016-09-08 22:38:03,346 ERROR datanode.DataNode (DataXceiver.java:run(278)) - xxxxxx06:50010:DataXceiver error processing unknown operation  src: /127.0.0.1:54501 dst: /127.0.0.1:50010
java.io.EOFException
	at java.io.DataInputStream.readShort(DataInputStream.java:315)
	at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
	at java.lang.Thread.run(Thread.java:745)

what is happening???? maybe y localhost ip????

Contributor

Can you show /etc/hosts file on node 6 and also hostsnames of the nodes. Can you try doing this: put your ambari-server hostname on top followed my master nodes in /etc/hosts. I had this issue earlier. It did work for me.

Super Collaborator

also not working

ip xxxxxx05 
ip xxxxxx07
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
ip xxxxxx01
ip xxxxxx02
ip xxxxxx03
ip xxxxxx04
ip xxxxxx06

Super Collaborator

also another curious thing is just the namenode CPU WIO i cant see

7503-snip20160908-3.png

@Roberto Sancho

I see all the responses above, but let me throw a crazy idea out there. Are you sure you don't have a filter for a specific host set in your Ambari UI?

Super Collaborator

not, i dont have any filter . thanks