Created on 10-26-2017 08:10 PM - edited 08-17-2019 10:26 AM
The Metrics Collector component is the central daemon that receives metrics from ALL the service sinks and monitors that sends metrics. The collector uses HBase as its store and phoenix as the data accessor layer.
In a high level, the metrics collector performs 2 operations related to scale in a continuos basis.
Scale problems occur in AMS when one or both of the above operations cannot happen smoothly. The 'load' on AMS is decided based on following factors
Either of the above can cause performance issues in AMS.
One or more of the following consequences can be seen on the cluster.
The above problems could occur because of a 2-3 underlying reasons.
|Underlying Problem||What it could cause||Fix / Workaround|
|Too many metrics (#4 from above)||It could cause ALL of the problems mentioned above.||#1 : Trying out config changes|
#2 : Reducing number of metrics
If the above config changes do not increase AMS stability, you can whitelist selected metrics or blacklist certain components' metrics that are causing the load issue.
AMS node has slow disk speed.
Disk is not able to keep up with high volume data.
|It can cause raw writes and aggregation problems.||
|Known issues around HBase normalier and FIFO compaction.|
Documented in Known Issues (#11 and #13)
|This can be identified in #5 in the above table.||Follow workaround steps in Known issue doc.|
|Configuration||Property||Description||Minimum Recommended values (Host Count => MB)|
|ams-site||phoenix.query.maxGlobalMemoryPercentage||Percentage of total heap memory used by Phoenix|
threads in the Metrics Collector API/Aggregator daemon.
|20 - 30, based on available memory. Default = 25.|
|ams-site||phoenix.spool.directory||Set directory for Phoenix spill files. (Client side)||Set this to different disk from hbase.rootdir dir if possible.|
|ams-hbase-site||phoenix.spool.directory||Set directory for Phoenix spill files. (Server side)||Set this to different disk from hbase.rootdir dir if possible.|
|ams-hbase-site||phoenix.query.spoolThresholdBytes||Threshold size in bytes after which results from parallelly|
executed query results are spooled to disk.
|Set this to higher value based on available memory.|
Default is 12 mb.