I am seeing lots of error in ambari-server.log file and metrics information is not displaced on Ambari dashboard (showing NA).
I have HDP3.0 installed on Linux 7 machines.
Here are the errors from ambari-server.log.
2018-09-18 17:06:30,361 ERROR [ambari-client-thread-35898] MetricsRequestHelper:112 - Error getting timeline metrics : Server returned HTTP response code: 403 for URL: http://worker2:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=master&appId=HOST
2018-09-18 17:06:30,380 ERROR [ambari-metrics-retrieval-service-thread-445] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:50070/jmx
2018-09-18 17:06:30,386 ERROR [ambari-metrics-retrieval-service-thread-439] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:8088/jmx
2018-09-18 17:06:30,388 ERROR [ambari-metrics-retrieval-service-thread-438] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:50070/jmx?get=Hadoop:service=NameNode,name=FSNamesystem::tag.HAState
2018-09-18 17:06:30,393 ERROR [ambari-metrics-retrieval-service-thread-441] URLStreamProvider:245 - Received HTTP 403 response from URL: http://worker1:19888/jmx
2018-09-18 17:06:30,406 ERROR [ambari-client-thread-35898] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:8088/ws/v1/cluster/info
2018-09-18 17:06:34,959 ERROR [ambari-client-thread-35892] URLStreamProvider:245 - Received HTTP 403 response from URL: http://worker2:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=master&appId=HOST
2018-09-18 17:06:34,960 ERROR [ambari-client-thread-35892] MetricsRequestHelper:112 - Error getting timeline metrics : Server returned HTTP response code: 403 for URL: http://worker2:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=master&appId=HOST
2018-09-18 17:06:34,980 ERROR [ambari-metrics-retrieval-service-thread-444] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:50070/jmx?get=Hadoop:service=NameNode,name=FSNamesystem::tag.HAState
Here troubleshooting steps output:
1. I am successfully able to open below URLs in web and download these url/files through wget.
wget http://master:8088/ws/v1/cluster/info
wget http://worker2:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=master&appId=HOST
2. AMS info in config file /etc/hadoop/conf/hadoop-metrics2.properties
$grep 'timeline.collector.host' /etc/hadoop/conf/hadoop-metrics2.properties | cut -d"=" -f2 | sort -n | uniq
datanode.sink.timeline.collector.hosts=worker2
namenode.sink.timeline.collector.hosts=worker2
resourcemanager.sink.timeline.collector.hosts=worker2
nodemanager.sink.timeline.collector.hosts=worker2
jobhistoryserver.sink.timeline.collector.hosts=worker2
journalnode.sink.timeline.collector.hosts=worker2
maptask.sink.timeline.collector.hosts=worker2
reducetask.sink.timeline.collector.hosts=worker2
applicationhistoryserver.sink.timeline.collector.hosts=worker2
$
3. Sink JAR is loaded on those Nodes/components
$ lsof -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid` | grep sink
java 6967 hdfs mem REG 249,0 5716944 1842669 /usr/lib/ambari-metrics-hadoop-sink/ambari-metrics-hadoop-sink-with-common-2.7.0.0.897.jar
java 6967 hdfs 322r REG 249,0 5716944 1842669 /usr/lib/ambari-metrics-hadoop-sink/ambari-metrics-hadoop-sink-with-common-2.7.0.0.897.jar
$
4. Ambari version
$ rpm -qa | grep ambari
ambari-metrics-grafana-2.7.0.0-897.x86_64
ambari-server-2.7.0.0-897.x86_64
ambari-metrics-hadoop-sink-2.7.0.0-897.x86_64
ambari-agent-2.7.0.0-897.x86_64
ambari-metrics-monitor-2.7.0.0-897.x86_64
$
5. ACTIVITY ANALYZER is going down after few minutes of restart.
$ grep -i "ACTIVITY_ANALYZER" ambari-server.log
2018-09-18 17:45:51,148 INFO [agent-report-processor-1] HeartbeatProcessor:581 - State of service component ACTIVITY_ANALYZER of service SMARTSENSE of cluster 104 has changed from STARTED to INSTALLED at host master according to STATUS_COMMAND report
$
