Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari Metrics not showing up and 503 error reported

SOLVED Go to solution
Highlighted

Ambari Metrics not showing up and 503 error reported

New Contributor

I have Ambari 2.7.3 set up on an Ubuntu 16.04 node and it has deployed a three-node cluster node.


Ambari Metrics was not working correctly, so I removed it and then re-installed it through the interface. This appeared to clear up several problems I was seeing in error logs but I'm still not able to see any metrics from any cluster nodes.

This seems to be connected to a problem showing up in the Ambari server log which keeps repeating the same error message:

2019-08-08 11:56:05,936 ERROR [ambari-client-thread-1655] MetricsRequestHelper:112 - Error getting timeline metrics : Server returned HTTP response code: 503 for URL: http://<metrics_collector_node_fqdn>:6188/ws/v1/timeline/metrics?metricNames=swap_free._avg&appId=HO....


So for some reason the node where the metric collector is running is reporting a 503 error. Yet if I look at the logs for the collector and also for each metrics monitor agent, they aren't reporting any errors. I only see one warning on the collector log that may or may not be relevant:

2019-08-08 11:53:41,682 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu Aug 08 11:53:41 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu Aug 08 11:50:00 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu Aug 08 11:50:00 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Last check point time: 1565290200000, lagBy: 221 seconds.
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu Aug 08 11:53:41 PDT 2019, startTime = Thu Aug 08 11:50:00 PDT 2019, endTime = Thu Aug 08 11:52:00 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Skipping aggregation for metric patterns : sdisk\_%,boottime
2019-08-08 11:53:42,771 INFO TimelineClusterAggregatorSecond: Saving 27764 metric aggregates.
2019-08-08 11:53:43,676 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu Aug 08 11:53:43 PDT 2019
2019-08-08 11:53:56,544 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Failed on insert records to store : null
2019-08-08 11:53:56,544 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Metric that cannot be stored : [default.General.hs2_avg_active_session_time,hiveserver2]{1565290341310=NaN, 1565290371310=NaN, 1565290401310=NaN, 1565290431311=NaN}
2019-08-08 11:53:56,545 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Failed on insert records to store : null
2019-08-08 11:53:56,545 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Metric that cannot be stored : [default.General.hs2_avg_open_session_time,hiveserver2]{1565290341310=NaN, 1565290371310=NaN, 1565290401310=NaN, 1565290431311=NaN}

Any ideas why I'm seeing a 503 error when my Ambari server tries to talk to the Amabi collector on another node?

Additional info:

  • Each node in my setup has a complete /etc/hosts file with all associated node FQDNs since I don't have DNS on my cluster network.
  • I am behind a proxy, but that info has been provided in the /var/lib/ambari-server/ambari-env.sh file per instructions as well as the default bash profile on each node. I also included my domain in the list of no-proxy arguments. I've had no issues with the proxy blocking comms with all my other services on the node where the collector is running.
  • I can copy the 503 error URL into Firefox on my Ambari server machine and it gets 200 OK with a JSON response. The headers that don't have any info suggesting the proxy was involved.
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Ambari Metrics not showing up and 503 error reported

New Contributor

Found the answer.


In my case I had my proxy settings defined in /etc/profile.d/proxy.sh so they were globally available.

But I also had them defined in /var/lib/ambari-server/ambari-env.sh as -D options under JVM args according to Ambari installation instructions. While this didn't cause problems getting status for most components of my deployed cluster, it caused "503 - Service Unavailable" errors in Ambari Metrics and the Files View.

Removing the -Dhttp.proxy settings from ambari-env.sh and re-starting solved the problem.

If you are setting your proxy info such that your ambari user account can get them from the environment, don't assign them again in the JVM_ARGS for Ambari Server!


1 REPLY 1

Re: Ambari Metrics not showing up and 503 error reported

New Contributor

Found the answer.


In my case I had my proxy settings defined in /etc/profile.d/proxy.sh so they were globally available.

But I also had them defined in /var/lib/ambari-server/ambari-env.sh as -D options under JVM args according to Ambari installation instructions. While this didn't cause problems getting status for most components of my deployed cluster, it caused "503 - Service Unavailable" errors in Ambari Metrics and the Files View.

Removing the -Dhttp.proxy settings from ambari-env.sh and re-starting solved the problem.

If you are setting your proxy info such that your ambari user account can get them from the environment, don't assign them again in the JVM_ARGS for Ambari Server!