It might be due to AMS tuning Specially the Heap settings might be incorrect.
Can you please let us know:
1. What is the cluster size (number of nodes in the cluster)
2. What is the Mode of AMS service Embedded or Distributed ? We can find it by looking at the property "timeline.metrics.service.operation.mode"
3. What is the current Heap Settings for "hbase_master_heapsize" , "metrics_collector_heapsize", "hbase_regionserver_heapsize"
The following doc provides a very good tuning for heap settings based on number of nodes in the cluster can you please try that:
Also if possible then can you please attach the output of the following API call which will help us in knowing how many metrics and what all metrics are being collected (just to findout if there are too many metrics).