Support Questions
Find answers, ask questions, and share your expertise

Error getting timeline metrics

Master Collaborator

I have HDP2.5 cluster up and running with no errors and everything in green except that metrics are not being displayed and I am getting following errors in the ambari-log.

I read around all the suggestions do not apply to me , e.g I have ntpd service enabled and iptables disabled on all nodes, also I have the proxy setup defined correctly and have no issues with wget or curl.

so what else could be the issue?

22 Jun 2017 17:30:39,801 ERROR [ambari-client-thread-98] MetricsRequestHelper:115 - Error getting timeline metrics : Server returned HTTP response code: 504 for URL: http://hadoop3.tolls.dot.state.fl.us:6188/ws/v1/timeline/metrics?metricNames=mem_total._avg&appId=HO...
22 Jun 2017 17:30:40,321 ERROR [ambari-client-thread-30] MetricsRequestHelper:115 - Error getting timeline metrics : Server returned HTTP response code: 504 for URL: http://hadoop3.tolls.dot.state.fl.us:6188/ws/v1/timeline/metrics?metricNames=cpu_wio&hostname=hadoop...
^C
You have new mail in /var/spool/mail/root
[root@hadoop1 ~]#
[root@hadoop1 ~]#
[root@hadoop1 ~]# curl http://hadoop3.tolls.dot.state.fl.us:6188/ws/v1/timeline/metrics?metricNames=mem_cached._avg&appId=H...
[1] 20007
[2] 20008
[3] 20009
[2]-  Done                    appId=HOST
[3]+  Done                    startTime=1498163394
[root@hadoop1 ~]# <!DOCTYPE html>
<html><head>
<title>504 Gateway Timeout</title>
</head><body style='font-family:Verdana'>
<h2><b>Gateway Timeout</b></h2>
<p>The requested URL couldn't be resolved</p>
</body></html>
^C
[1]+  Done                    curl http://hadoop3.tolls.dot.state.fl.us:6188/ws/v1/timeline/metrics?metricNames=mem_cached._avg
[root@hadoop1 ~]#
6 REPLIES 6

Super Mentor

@Sami Ahmad

504 indicates some proxy issue here.

You can configure ambari to use the proxy host or not use the proxy to reach to AMS collector on port 6188 using the following approaches:

  1. Add proxy settings to the following script: /var/lib/ambari-server/ambari-env.sh.
    -Dhttp.proxyHost=<yourProxyHost> -Dhttp.proxyPort=<yourProxyPort>
  2. Optionally, to prevent some host names from accessing the proxy server, define the list of excluded hosts, as follows:
    -Dhttp.nonProxyHosts=<pipe|separated|list|of|hosts> 

These settings you can add inside the "AMBARI_JVM_ARGS" of your /var/lib/ambari-server/ambari-env.sh script.

.

.

Master Collaborator

ambari-server is running on hadoop1 , and the proxy is setup in the ambari-env.sh.

AMBARI_PASSHPHRASE="DEV"
export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/k
rb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dhttp.proxyHost=dotatofwproxy.tolls.dot.state.fl.us -Dhttp.proxyPort=8080'

The error I am getting is on hadoop3 , you mean set up the proxy settings there?

also if it was true proxy error how did the cluster got installed properly ?

Super Mentor

@Sami Ahmad

Based on your some older HCC threads i think you have excluded few hosts using "no_proxy" option which means you actually do not want those requests to pass through the proxy.

export no_proxy=".tolls.dot.state.fl.us,hadoop1,hadoop2,hadoop3,hadoop4,hadoop5"

https://community.hortonworks.com/questions/107933/error-loading-dbconnectionverificationjar.html#an...

So if you want that ambari also should NOT request the AMS collector URL (http://hadoop3.tolls.dot.state.fl.us:6188) Via Proxy then you can use the nonProxyHosts option.

-Dhttp.nonProxyHosts=<pipe|separated|list|of|hosts

.

Master Collaborator

hi Jay

and the no_proxy is defined , that's how I could get the Hivemeta store to work.

I have a feeling from other posts on this site that the error I am getting might be misleading and real issue is something else.

could this be some parameter or memory issue?

[root@hadoop1 ~]# echo $no_proxy
.tolls.dot.state.fl.us,hadoop1,hadoop2,hadoop3,hadoop4,hadoop5

Master Collaborator

has anyone got the Ambari metrics to work behind a proxy server ?

Contributor

Ambari server was getting HTTP 502 trying to query timeline metrics - this fixed it for us behind corporate firewall. Thanks!

; ;