question Re: why i can't see the ambari-metrics, my ambari metrics service is installed in Archives of Support Questions (Read Only)

why i can't see the ambari-metrics, my ambari metrics service is installed

yadir97_ya — Sun, 18 Aug 2019 09:48:49 GMT

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

jsensharma — Thu, 30 Aug 2018 05:01:33 GMT

You seems to be looking at the Ambari Dashboard.

Can you please click on the individual service like "HDFS" and then check the widgets in "Summary" section bottom.

Also every service does not provide the metrics via Widgets like Zookeeper/ SmartSense/Slider so you will not see metrics widgets for them.

Also you can check the metrics details using Grafana UI which actually pulls the metrics details from Ambari Metrics Collector.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/grafana_dashboard_reference.html

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

yadir97_ya — Sun, 18 Aug 2019 09:48:41 GMT

@Jay Kumar SenSharma

i check this:

->Ambari metrics dashboard. 

->cat /var/log/ambari-metrics-collector/gc.log-201808291552

->cat /var/log/ambari-metrics-collector/hbase-ams-master-maestro.hdp.com.log |grep -i pause

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

jsensharma — Thu, 30 Aug 2018 05:43:09 GMT

@yadir Aguilar

Few things:
The screenshot "memory.png" is showing the GC log of HMaster process and not for the Ambari Metrics Collector process.
The GC log file for AMS will be this one "/var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx"
.

The "pause.png" screenshot shows some INFO messages and the JVMUtil pause messages are mostly concerned when they are WARN level. INFO messages about the pause can keep appearing in the logs. So you should do this to actually see the Pauses which needs to be reviewed.

# cat /var/log/ambari-metrics-collector/hbase-ams-master-xxxxxxxxx.log  | grep -i pause | grep -i WARN

General Recommendation (trial basis)
Also if you are running AMS in Embedded Mode default settigns then i will suggest you to try increasing the AMS collector heap size from default 512 MB to a bit high value lik2 1G to 2G and then restart the AMS Collector again. Similarly try increasing the HMaster memory to something around 2G to 4G and see if it helps. The parameters can be found here: in ambari UI

"ams-env" --> "metrics_collector_heapsize" 
"ams-hbase-env" --> "hbase_master_heapsize"

For More accurate Tuning Recommendations (data needed)
For more accurate tuning of AMS service you may need to collect some additional details like the output fo the following API calls and the current memory usage:

1. The JSON response from the following API calls to see the uni queue metrics and hosts, to understand the load on AMS.

http://<ams-host>:6188/ws/v1/timeline/metrics/metadata
http://<ams-host>:6188/ws/v1/timeline/metrics/hosts

2. Current memory usages (either complete GC logs HMaster /var/log/ambari-metrics-collector/gc.log-xxxxxx and for AMS collector GC log : /var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx )

3. The output of the AMS HMaster jmx call: AMS HBase jmx Snapshot -

http://<METRICS_COLLECTOR_HOST>:61310/jmx

4. The AMS HMaster UI output to understand the "Region Count" and "StoreFile Count"

http://<METRICS_COLLECTOR_HOST>:61310

5. AMS collector and HMaster GC complete logs.

6. AMS Configurations:

# tar czhfv /tmp/ams_collector_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ambari-metrics-collector/ 
# tar czhfv /tmp/ams_hmaster_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ams-hbase/

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

jsensharma — Thu, 30 Aug 2018 05:45:03 GMT

@yadir Aguilar

There are some very useful Ambari Metrics Tuning Articles / docs available here which you might want to refer:

1. https://cwiki.apache.org/confluence/display/AMBARI/Configurations+-+Tuning

2. https://community.hortonworks.com/content/supportkb/208353/troubleshooting-ambari-metrics-ams.html

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

yadir97_ya — Fri, 31 Aug 2018 00:02:44 GMT

@Jay Kumar SenSharma

can be also a ram memory problem?

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

jsensharma — Fri, 31 Aug 2018 06:21:30 GMT

Yes, Insufficient RAM can be a problem. However this can be investigated/confirmed based on the data requested above.