Ambari Metrics not showing up and 503 error reported

[#1]

I have Ambari 2.7.3 set up on an Ubuntu 16.04 node and it has deployed a three-node cluster node.

Ambari Metrics was not working correctly, so I removed it and then re-installed it through the interface. This appeared to clear up several problems I was seeing in error logs but I'm still not able to see any metrics from any cluster nodes.

This seems to be connected to a problem showing up in the Ambari server log which keeps repeating the same error message:

2019-08-08 11:56:05,936 ERROR [ambari-client-thread-1655] MetricsRequestHelper:112 - Error getting timeline metrics : Server returned HTTP response code: 503 for URL: http://<metrics_collector_node_fqdn>:6188/ws/v1/timeline/metrics?metricNames=swap_free._avg&appId=HO....

So for some reason the node where the metric collector is running is reporting a 503 error. Yet if I look at the logs for the collector and also for each metrics monitor agent, they aren't reporting any errors. I only see one warning on the collector log that may or may not be relevant:

2019-08-08 11:53:41,682 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu Aug 08 11:53:41 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu Aug 08 11:50:00 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu Aug 08 11:50:00 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Last check point time: 1565290200000, lagBy: 221 seconds.
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu Aug 08 11:53:41 PDT 2019, startTime = Thu Aug 08 11:50:00 PDT 2019, endTime = Thu Aug 08 11:52:00 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Skipping aggregation for metric patterns : sdisk\_%,boottime
2019-08-08 11:53:42,771 INFO TimelineClusterAggregatorSecond: Saving 27764 metric aggregates.
2019-08-08 11:53:43,676 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu Aug 08 11:53:43 PDT 2019
2019-08-08 11:53:56,544 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Failed on insert records to store : null
2019-08-08 11:53:56,544 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Metric that cannot be stored : [default.General.hs2_avg_active_session_time,hiveserver2]{1565290341310=NaN, 1565290371310=NaN, 1565290401310=NaN, 1565290431311=NaN}
2019-08-08 11:53:56,545 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Failed on insert records to store : null
2019-08-08 11:53:56,545 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Metric that cannot be stored : [default.General.hs2_avg_open_session_time,hiveserver2]{1565290341310=NaN, 1565290371310=NaN, 1565290401310=NaN, 1565290431311=NaN}

Any ideas why I'm seeing a 503 error when my Ambari server tries to talk to the Amabi collector on another node?

Additional info:

• Each node in my setup has a complete /etc/hosts file with all associated node FQDNs since I don't have DNS on my cluster network.
• I am behind a proxy, but that info has been provided in the /var/lib/ambari-server/ambari-env.sh file per instructions as well as the default bash profile on each node. I also included my domain in the list of no-proxy arguments. I've had no issues with the proxy blocking comms with all my other services on the node where the collector is running.
• I can copy the 503 error URL into Firefox on my Ambari server machine and it gets 200 OK with a JSON response. The headers that don't have any info suggesting the proxy was involved.

[#2 · Accepted solution]

Found the answer.

In my case I had my proxy settings defined in /etc/profile.d/proxy.sh so they were globally available.

But I also had them defined in /var/lib/ambari-server/ambari-env.sh as -D options under JVM args according to Ambari installation instructions. While this didn't cause problems getting status for most components of my deployed cluster, it caused "503 - Service Unavailable" errors in Ambari Metrics and the Files View.

Removing the -Dhttp.proxy settings from ambari-env.sh and re-starting solved the problem.

If you are setting your proxy info such that your ambari user account can get them from the environment, don't assign them again in the JVM_ARGS for Ambari Server!