I have HDP2.5 cluster up and running with no errors and everything in green except that metrics are not being displayed and I am getting following errors in the ambari-log.
I read around all the suggestions do not apply to me , e.g I have ntpd service enabled and iptables disabled on all nodes, also I have the proxy setup defined correctly and have no issues with wget or curl.
so what else could be the issue?
22 Jun 2017 17:30:39,801 ERROR [ambari-client-thread-98] MetricsRequestHelper:115 - Error getting timeline metrics : Server returned HTTP response code: 504 for URL: http://hadoop3.tolls.dot.state.fl.us:6188/ws/v1/timeline/metrics?metricNames=mem_total._avg&appId=HO... 22 Jun 2017 17:30:40,321 ERROR [ambari-client-thread-30] MetricsRequestHelper:115 - Error getting timeline metrics : Server returned HTTP response code: 504 for URL: http://hadoop3.tolls.dot.state.fl.us:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=hadoop... ^C You have new mail in /var/spool/mail/root [root@hadoop1 ~]# [root@hadoop1 ~]# [root@hadoop1 ~]# curl http://hadoop3.tolls.dot.state.fl.us:6188/ws/v1/timeline/metrics?metricNames=mem_cached._avg&appId=H...  20007  20008  20009 - Done appId=HOST + Done startTime=1498163394 [root@hadoop1 ~]# <!DOCTYPE html> <html><head> <title>504 Gateway Timeout</title> </head><body style='font-family:Verdana'> <h2><b>Gateway Timeout</b></h2> <p>The requested URL couldn't be resolved</p> </body></html> ^C + Done curl http://hadoop3.tolls.dot.state.fl.us:6188/ws/v1/timeline/metrics?metricNames=mem_cached._avg [root@hadoop1 ~]#
504 indicates some proxy issue here.
You can configure ambari to use the proxy host or not use the proxy to reach to AMS collector on port 6188 using the following approaches:
These settings you can add inside the "AMBARI_JVM_ARGS" of your
ambari-server is running on hadoop1 , and the proxy is setup in the ambari-env.sh.
AMBARI_PASSHPHRASE="DEV" export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/k rb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dhttp.proxyHost=dotatofwproxy.tolls.dot.state.fl.us -Dhttp.proxyPort=8080'
The error I am getting is on hadoop3 , you mean set up the proxy settings there?
also if it was true proxy error how did the cluster got installed properly ?
Based on your some older HCC threads i think you have excluded few hosts using "no_proxy" option which means you actually do not want those requests to pass through the proxy.
and the no_proxy is defined , that's how I could get the Hivemeta store to work.
I have a feeling from other posts on this site that the error I am getting might be misleading and real issue is something else.
could this be some parameter or memory issue?
[root@hadoop1 ~]# echo $no_proxy .tolls.dot.state.fl.us,hadoop1,hadoop2,hadoop3,hadoop4,hadoop5