Hosts and ambari metrics are not visible

[#1]

batch4-ambarimetrics.jpgbatch4-hosts-metrics.jpg

I have migrated metrics collector data and changed metrics collector embedded mode to distributed for that I referred

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-operations/content/ams_metrics_co...

link . There are no errors in logs. Everything seems to be fine except I cannot see Ambari Metrics and Hosts Metrics. Attached the screenshot.

Thanks!!

[#2]

@Manisha Pande

Which version of ambari is it? Please run the following command on few hosts including the collectos hosts to findout if the ambari server and ams binary versions are same.

# rpm -qa | grep ambari

- Can you please check if you have the ambari-metrics monitors are pointing to correct hosts?

# grep 'metrics_servers'  /etc/ambari-metrics-monitor/conf/metric_monitor.ini 
# grep 'port'  /etc/ambari-metrics-monitor/conf/metric_monitor.ini 

- Is that Host name correct and reachable from other hosts? Also are you able to do telnet to the AMS collector host & ports from monitor machines?

# nc -v  $AMS_COLLECTOR_HOSTNAME  6188
(OR)
# telnet  $AMS_COLLECTOR_HOSTNAME  6188

4. Have you recently performed Ambari Server Upgrade? If yes then have you also operformed the Ambari Metrics Service Upgrade (Post Upgrade Steps as well?) https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-upgrade/content/upgrade_ambari_me...

5. Can you please share the metrics monitor logs from the hosts which does not show host metrics?

 /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out 

6. If possible then can you also share the AMS collector logs (although you mentioend that there are no errors ... it will be good to review it if there are some strange warnings).

.

[#3]

@Jay Kumar SenSharma

Please find the requested details

1. ambari server and ams binary versions

$ rpm -qa | grep ambari
ambari-metrics-monitor-2.5.0.3-7.x86_64
ambari-metrics-grafana-2.5.0.3-7.x86_64
ambari-metrics-hadoop-sink-2.5.0.3-7.x86_64
ambari-agent-2.5.0.3-7.x86_64
ambari-metrics-collector-2.5.0.3-7.x86_64

2. ambari-metrics monitors are pointing to correct servers

metrics_servers = ics024045196.ics-eu-1.asml.com,ics024045188.ics-eu-1.asml.com

3. $ nc -v ics024045188.ics-eu-1.asml.com 6188
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to 172.24.45.188:6188.

$ telnet ics024045188.ics-eu-1.asml.com 6188
Trying 172.24.45.188...
Connected to ics024045188.ics-eu-1.asml.com.

4. No, I didnt upgarded amabri metrics

5. ambari-monitor logs

2017-11-22 11:24:04,908 [INFO] host_info.py:352 - Cached hostname: ics024045188.ics-eu-1.asml.com
2017-11-22 11:24:04,909 [INFO] controller.py:101 - Adding event to cache, all : {u'metrics': [{u'value_threshold': u'128', u'name': u'bytes_out'}], u'collect_every': u'10'}
2017-11-22 11:24:04,909 [INFO] main.py:65 - Starting Server RPC Thread: /usr/lib/python2.6/site-packages/resource_monitoring/main.py start
2017-11-22 11:24:04,909 [INFO] controller.py:55 - Running Controller thread: Thread-1
2017-11-22 11:24:04,909 [INFO] emitter.py:60 - Running Emitter thread: Thread-2
2017-11-22 11:24:04,910 [INFO] emitter.py:79 - Nothing to emit, resume waiting.
2017-11-22 11:25:04,912 [INFO] emitter.py:152 - Calculated collector shard based on hostname : ics024045196.ics-eu-1.asml.com
2017-11-22 11:26:04,919 [INFO] emitter.py:152 - Calculated collector shard based on hostname : ics024045196.ics-eu-1.asml.com
2017-11-22 11:27:04,930 [INFO] emitter.py:152 - Calculated collector shard based on hostname : ics024045196.ics-eu-1.asml.com
2017-11-22 11:28:04,939 [INFO] emitter.py:152 - Calculated collector shard based on hostname : ics024045196.ics-eu-1.asml.com
2017-11-22 11:29:04,953 [INFO] emitter.py:152 - Calculated collector shard based on hostname : ics024045196.ics-eu-1.asml.com
2017-11-22 11:30:04,968 [INFO] emitter.py:152 - Calculated collector shard based on hostname : ics024045196.ics-eu-1.asml.com
2017-11-22 11:31:04,994 [INFO] emitter.py:152 - Calculated collector shard based on hostname : ics024045196.ics-eu-1.asml.com
2017-11-22 11:32:05,039 [INFO] emitter.py:152 - Calculated collector shard based on hostname : ics024045196.ics-eu-1.asml.com
Server out at: /var/log/ambari-metrics-host-monitoring/ambari-metrics-host-monitoring.out
Server log at: /var/log/ambari-metrics-host-monitoring/ambari-metrics-host-monitoring.log

[#4]

same problem here ,everything is good except host metric and ambari-metric itself's metric

[#5]

@Manisha Pande

Can you attach the below logs

• /var/log/ambari-metrics-host-monitoring/ambari-metrics-host-monitoring.log
• /var/log/ambari-metrics-host-monitoring/ambari-metrics-host-monitoring.out

Step-by-step guide to resolve AMS

1. Using Ambari
   1. Set AMS to maintenance
   2. Stop AMS from Ambari
   3. Identify the following from the AMS Configs screen
      1. 'Metrics Service operation mode' (embedded or distributed)
      2. hbase.rootdir
      3. hbase.zookeeper.property.dataDir
2. AMS data would be stored in 'hbase.rootdir' identified above. Backup and remove the AMS data.
   1. If the Metrics Service operation mode
      1. is 'embedded', then the data is stored in OS files. Use regular OS commands to backup and remove the files in hbase.rootdir
      2. is 'distributed', then the data is stored in HDFS. Use 'hdfs dfs' commands to backup and remove the files in hbase.rootdir
3. Remove the AMS zookeeper data by backing up and removing the contents of 'hbase.tmp.dir'/zookeeper
4. Remove any Phoenix spool files from 'hbase.tmp.dir'/phoenix-spool folder
5. Restart AMS using Ambari

Hope that help please revert