Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

ambari metrics collector going down

avatar

Ambari metrics collector is going down because of lack of thread pools. How to increase the thread pool size for ambari metrics hbase. we are running hbase for metrics in distributed mode. Collector gows down within 5 minutes after restart.

1 ACCEPTED SOLUTION

avatar
Super Collaborator

@ARUN

Based on the logs, it seems one or more components are flooding the system with too many metrics. It could be the cluster HBase Service.

Can you check if the last 2 lines in the files mentioned in https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/_enabling... are not commented out?

The last 2 lines should look like this (and should not be commented out).

*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter

hbase.*.source.filter.exclude=*Regions*

Restart HBase Service after these changes.

Also, for a 30 node cluster, AMS should work fine with embedded mode, writing data to local disk. Your cluster AMS is configured to distributed mode where AMS HBase writes to cluster HDFS. Do you have a local datanode on Metrics collector host?

View solution in original post

11 REPLIES 11

avatar

@Aravindan Vijayan, these 2 lines are not present in any of the 2 files mentiones in the url you gave. That means this is equivalent to getting commented out. so metrics at region level too are getting flooded. YEs the metrics collector host is cohosted with a datanode. but we are planning to move it to a dedicated admin host.

avatar
Master Mentor

@ARUN

Here the instruction is to disable (exclude) HBase per region metrics to avoid data flooding.

That can be done by explicitly adding the following lines to the end of the file:

*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter
hbase.*.source.filter.exclude=*Regions*

.