Ambari Metric Server basic tuning

[#1]

1. If using Ambari versions before 2.1.2, Disable HBase per region metrics. On Ambari server host. Edit the following files under /var/lib/ambari-server/resources/:

common-services/HBASE/0.96.0.2.0/package/templates/HBASE/hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
common-services/HBASE/0.96.0.2.0/package/templates/HBASE/hadoop-metrics2-hbase.properties-GANGLIA-RS.j2

and add the following lines at the end before '% endif %}':

*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter
hbase.*.source.filter.exclude=*Regions*

Do a rolling restart of the HBase RegionServers. Note: This does not disable RS metrics. It just disables the per region / per table metrics collected at Region level. This is disabled by default from Ambari 2.1.2. 2. Tune AMS configs: Find out the Heap available to AMS collector host. Change the following settings based on available memory:

ams-hbase-env_ :: hbase_master_heapsize = 8192m (Or 16384m if available)
ams-hbase-env_ :: hbase_master_xmn_size = 1024m
ams-hbase-env_ :: regionserver_xmn_size = 1024m
ams-hbase-site :: phoenix.query.spoolThresholdBytes= 25165824 (24 MB from 12 MB)

3. AMS data storage If using embedded mode, change the write paths for

ams-hbase-site :: hbase.rootdir 

ams-hbase-site :: hbase.tmp.dir

so that its placed in the fastest possible disk. Also it's better to keep hbase.tmp.dir in a location different from hbase.rootdir

After completing the above, stop AMS from Ambari. Once stopped ensure that the process are stopped by doing a

ps -aux | grep ams

If the process are still around, kill the same and clean up /var/run/ambari-metrics-collector/*.pid file.

Now restart AMS services using Ambari.