Support Questions

arunpoy · ‎12-20-2016

Ambari metrics collector is going down because of lack of thread pools. How to increase the thread pool size for ambari metrics hbase. we are running hbase for metrics in distributed mode. Collector gows down within 5 minutes after restart.

avijayan · ‎01-04-2017

@ARUN

Based on the logs, it seems one or more components are flooding the system with too many metrics. It could be the cluster HBase Service.

Can you check if the last 2 lines in the files mentioned in https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/_enabling... are not commented out?

The last 2 lines should look like this (and should not be commented out).

*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter

hbase.*.source.filter.exclude=*Regions*

Restart HBase Service after these changes.

Also, for a 30 node cluster, AMS should work fine with embedded mode, writing data to local disk. Your cluster AMS is configured to distributed mode where AMS HBase writes to cluster HDFS. Do you have a local datanode on Metrics collector host?

View solution in original post

arunpoy · ‎01-05-2017

@Aravindan Vijayan, these 2 lines are not present in any of the 2 files mentiones in the url you gave. That means this is equivalent to getting commented out. so metrics at region level too are getting flooded. YEs the metrics collector host is cohosted with a datanode. but we are planning to move it to a dedicated admin host.

jsensharma · ‎01-05-2017

@ARUN

Here the instruction is to disable (exclude) HBase per region metrics to avoid data flooding.

That can be done by explicitly adding the following lines to the end of the file:

*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter
hbase.*.source.filter.exclude=*Regions*

.

Cloudera Community

Support Questions

ambari metrics collector going down