Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ambari metrics using 100% cpu

avatar
Contributor

I have a HDP 2.3.0 cluster with 4 nodes. I noticed this process was consuming 100% cpu on my NameNode:

ams       5386  223  5.0 3596120 1666616 ?     Sl   13:50  46:56 /usr/jdk64/jdk1.8.0_40/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -Xmx1536m -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/ambari-metrics-collector/hs_err_pid%p.log -Djava.io.tmpdir=/opt/var/lib/ambari-metrics-collector/hbase-tmp -Djava.library.path=/usr/lib/ams-hbase/lib/hadoop-native/ -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/ambari-metrics-collector/gc.log-201512161350 -Xms1536m -Xmx1536m -Xmn256m -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -Dhbase.log.dir=/var/log/ambari-metrics-collector -Dhbase.log.file=hbase-ams-master-NPAA1809.petrobras.biz.log -Dhbase.home.dir=/usr/lib/ams-hbase/bin/.. -Dhbase.id.str=ams -Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster start

But my cluster does not have Hbase installed. Then I simply killed this process, but Ambari metrics went down together. After I restarted Ambari metrics, this process continues trying to executing and always consuming a lot of CPU. Here is a print of Ambari Metrics durint time of my actions.

882-ambari.png

How can I configure AMS to stop trying to monitor HBase?

1 ACCEPTED SOLUTION

avatar
Super Collaborator

What is the AMS operation mode ("Metrics Service operation mode" config) ?

Please share all the memory configs of AMS along with the memory in the node. The following configs will be useful.

In ams-env - metrics_collector_heapsize

In ams-hbase-env - hbase_regionserver_heapsize,hbase_master_maxperm_size, hbase_master_xmn_size, hbase_regionserver_heapsize, regionserver_xmn_size

and the output of "free -m"

View solution in original post

15 REPLIES 15

avatar
Contributor

Here is my config files for ambari-metrics-collector

ambari-metrics-collector.tar.gz

avatar

@Vitor Batista Can you try setting server.timeline.metrics.cache.disabled = true in ambari.properties file and restart ambari-server for a try? I don't think you have enough resources on this machine.

avatar
Contributor

I tried, but nothing changed. Where should I configure the collector interval. I don't need a "real-time" monitoring. Maybe 1 measurement per minute is enough.

avatar

@Vitor Batista As @Guilherme Braccialli rightly mentioned Ambari Metrics monitor all services / hosts and it consumes resources to run. For small test environments you can stop Ambari Metrics with few cpu, you can stop ambari metrics, it wont affect other components. Our Sandbox, for example, does not have Ambari Metrics started by default. This VM size you have is very small for so many components.

avatar
Super Collaborator

What is the AMS operation mode ("Metrics Service operation mode" config) ?

Please share all the memory configs of AMS along with the memory in the node. The following configs will be useful.

In ams-env - metrics_collector_heapsize

In ams-hbase-env - hbase_regionserver_heapsize,hbase_master_maxperm_size, hbase_master_xmn_size, hbase_regionserver_heapsize, regionserver_xmn_size

and the output of "free -m"

avatar
Master Mentor

@Vitor Batista are you still having issues with this? Can you accept the best answer or post your solution?