Ambari Metrics not showing up and 503 error reported
I have Ambari 2.7.3 set up on an Ubuntu 16.04 node and it has deployed a three-node cluster node.
Ambari Metrics was not working correctly, so I removed it and then re-installed it through the interface. This appeared to clear up several problems I was seeing in error logs but I'm still not able to see any metrics from any cluster nodes.
This seems to be connected to a problem showing up in the Ambari server log which keeps repeating the same error message:
2019-08-08 11:56:05,936 ERROR [ambari-client-thread-1655] MetricsRequestHelper:112 - Error getting timeline metrics : Server returned HTTP response code: 503 for URL: http://<metrics_collector_node_fqdn>:6188/ws/v1/timeline/metrics?metricNames=swap_free._avg&appId=HOST&startTime=1565286966&endTime=1565290566.
So for some reason the node where the metric collector is running is reporting a 503 error. Yet if I look at the logs for the collector and also for each metrics monitor agent, they aren't reporting any errors. I only see one warning on the collector log that may or may not be relevant:
2019-08-08 11:53:41,682 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Thu Aug 08 11:53:41 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Thu Aug 08 11:50:00 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Thu Aug 08 11:50:00 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Last check point time: 1565290200000, lagBy: 221 seconds.
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Thu Aug 08 11:53:41 PDT 2019, startTime = Thu Aug 08 11:50:00 PDT 2019, endTime = Thu Aug 08 11:52:00 PDT 2019
2019-08-08 11:53:41,684 INFO TimelineClusterAggregatorSecond: Skipping aggregation for metric patterns : sdisk\_%,boottime
2019-08-08 11:53:42,771 INFO TimelineClusterAggregatorSecond: Saving 27764 metric aggregates.
2019-08-08 11:53:43,676 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Thu Aug 08 11:53:43 PDT 2019
2019-08-08 11:53:56,544 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Failed on insert records to store : null
2019-08-08 11:53:56,544 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Metric that cannot be stored : [default.General.hs2_avg_active_session_time,hiveserver2]{1565290341310=NaN, 1565290371310=NaN, 1565290401310=NaN, 1565290431311=NaN}
2019-08-08 11:53:56,545 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Failed on insert records to store : null
2019-08-08 11:53:56,545 WARN org.apache.ambari.metrics.core.timeline.PhoenixHBaseAccessor: Metric that cannot be stored : [default.General.hs2_avg_open_session_time,hiveserver2]{1565290341310=NaN, 1565290371310=NaN, 1565290401310=NaN, 1565290431311=NaN}
Any ideas why I'm seeing a 503 error when my Ambari server tries to talk to the Amabi collector on another node?
Additional info:
- Each node in my setup has a complete /etc/hosts file with all associated node FQDNs since I don't have DNS on my cluster network.
- I am behind a proxy, but that info has been provided in the /var/lib/ambari-server/ambari-env.sh file per instructions as well as the default bash profile on each node. I also included my domain in the list of no-proxy arguments. I've had no issues with the proxy blocking comms with all my other services on the node where the collector is running.
- I can copy the 503 error URL into Firefox on my Ambari server machine and it gets 200 OK with a JSON response. The headers that don't have any info suggesting the proxy was involved.
