- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
ambari metrics collector going down
- Labels:
-
Apache Ambari
-
Apache HBase
Created ‎12-20-2016 04:47 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ambari metrics collector is going down because of lack of thread pools. How to increase the thread pool size for ambari metrics hbase. we are running hbase for metrics in distributed mode. Collector gows down within 5 minutes after restart.
Created ‎01-04-2017 11:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Based on the logs, it seems one or more components are flooding the system with too many metrics. It could be the cluster HBase Service.
Can you check if the last 2 lines in the files mentioned in https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/_enabling... are not commented out?
The last 2 lines should look like this (and should not be commented out).
*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter hbase.*.source.filter.exclude=*Regions*
Restart HBase Service after these changes.
Also, for a 30 node cluster, AMS should work fine with embedded mode, writing data to local disk. Your cluster AMS is configured to distributed mode where AMS HBase writes to cluster HDFS. Do you have a local datanode on Metrics collector host?
Created ‎01-05-2017 05:47 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Aravindan Vijayan, these 2 lines are not present in any of the 2 files mentiones in the url you gave. That means this is equivalent to getting commented out. so metrics at region level too are getting flooded. YEs the metrics collector host is cohosted with a datanode. but we are planning to move it to a dedicated admin host.
Created ‎01-05-2017 06:04 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here the instruction is to disable (exclude) HBase per region metrics to avoid data flooding.
That can be done by explicitly adding the following lines to the end of the file:
*.source.filter.class=org.apache.hadoop.metrics2.filter.GlobFilter hbase.*.source.filter.exclude=*Regions*
.

- « Previous
-
- 1
- 2
- Next »