Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP3.0 Ambari-Metrics : ambari-metrics-retrieval-service-thread ERROR Received HTTP 403 response from URL

HDP3.0 Ambari-Metrics : ambari-metrics-retrieval-service-thread ERROR Received HTTP 403 response from URL

Contributor

I am seeing lots of error in ambari-server.log file and metrics information is not displaced on Ambari dashboard (showing NA).

I have HDP3.0 installed on Linux 7 machines.

Here are the errors from ambari-server.log.

2018-09-18 17:06:30,361 ERROR [ambari-client-thread-35898] MetricsRequestHelper:112 - Error getting timeline metrics : Server returned HTTP response code: 403 for URL: http://worker2:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=master&appId=HOST
2018-09-18 17:06:30,380 ERROR [ambari-metrics-retrieval-service-thread-445] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:50070/jmx
2018-09-18 17:06:30,386 ERROR [ambari-metrics-retrieval-service-thread-439] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:8088/jmx
2018-09-18 17:06:30,388 ERROR [ambari-metrics-retrieval-service-thread-438] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:50070/jmx?get=Hadoop:service=NameNode,name=FSNamesystem::tag.HAState
2018-09-18 17:06:30,393 ERROR [ambari-metrics-retrieval-service-thread-441] URLStreamProvider:245 - Received HTTP 403 response from URL: http://worker1:19888/jmx
2018-09-18 17:06:30,406 ERROR [ambari-client-thread-35898] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:8088/ws/v1/cluster/info
2018-09-18 17:06:34,959 ERROR [ambari-client-thread-35892] URLStreamProvider:245 - Received HTTP 403 response from URL: http://worker2:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=master&appId=HOST
2018-09-18 17:06:34,960 ERROR [ambari-client-thread-35892] MetricsRequestHelper:112 - Error getting timeline metrics : Server returned HTTP response code: 403 for URL: http://worker2:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=master&appId=HOST
2018-09-18 17:06:34,980 ERROR [ambari-metrics-retrieval-service-thread-444] URLStreamProvider:245 - Received HTTP 403 response from URL: http://master:50070/jmx?get=Hadoop:service=NameNode,name=FSNamesystem::tag.HAState
Here troubleshooting steps output:

1. I am successfully able to open below URLs in web and download these url/files through wget.

wget http://master:8088/ws/v1/cluster/info
wget http://worker2:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=master&appId=HOST

2. AMS info in config file /etc/hadoop/conf/hadoop-metrics2.properties

$grep 'timeline.collector.host'  /etc/hadoop/conf/hadoop-metrics2.properties | cut -d"=" -f2 | sort -n | uniq

datanode.sink.timeline.collector.hosts=worker2
namenode.sink.timeline.collector.hosts=worker2
resourcemanager.sink.timeline.collector.hosts=worker2
nodemanager.sink.timeline.collector.hosts=worker2
jobhistoryserver.sink.timeline.collector.hosts=worker2
journalnode.sink.timeline.collector.hosts=worker2
maptask.sink.timeline.collector.hosts=worker2
reducetask.sink.timeline.collector.hosts=worker2
applicationhistoryserver.sink.timeline.collector.hosts=worker2
$

3. Sink JAR is loaded on those Nodes/components

$ lsof -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid`  | grep sink
java    6967 hdfs  mem       REG              249,0   5716944  1842669 /usr/lib/ambari-metrics-hadoop-sink/ambari-metrics-hadoop-sink-with-common-2.7.0.0.897.jar
java    6967 hdfs  322r      REG              249,0   5716944  1842669 /usr/lib/ambari-metrics-hadoop-sink/ambari-metrics-hadoop-sink-with-common-2.7.0.0.897.jar
$

4. Ambari version

$ rpm -qa | grep ambari
ambari-metrics-grafana-2.7.0.0-897.x86_64
ambari-server-2.7.0.0-897.x86_64
ambari-metrics-hadoop-sink-2.7.0.0-897.x86_64
ambari-agent-2.7.0.0-897.x86_64
ambari-metrics-monitor-2.7.0.0-897.x86_64
$

5. ACTIVITY ANALYZER is going down after few minutes of restart.

$ grep -i "ACTIVITY_ANALYZER" ambari-server.log
2018-09-18 17:45:51,148  INFO [agent-report-processor-1] HeartbeatProcessor:581 - State of service component ACTIVITY_ANALYZER of service SMARTSENSE of cluster 104 has changed from STARTED to INSTALLED at host master according to STATUS_COMMAND report
$