Created on 08-29-2018 09:51 PM - edited 08-18-2019 02:48 AM
Created 08-29-2018 10:43 PM
Few things:
The screenshot "memory.png" is showing the GC log of HMaster process and not for the Ambari Metrics Collector process.
The GC log file for AMS will be this one "/var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx"
.
The "pause.png" screenshot shows some INFO messages and the JVMUtil pause messages are mostly concerned when they are WARN level. INFO messages about the pause can keep appearing in the logs. So you should do this to actually see the Pauses which needs to be reviewed.
# cat /var/log/ambari-metrics-collector/hbase-ams-master-xxxxxxxxx.log | grep -i pause | grep -i WARN
General Recommendation (trial basis)
Also if you are running AMS in Embedded Mode default settigns then i will suggest you to try increasing the AMS collector heap size from default 512 MB to a bit high value lik2 1G to 2G and then restart the AMS Collector again. Similarly try increasing the HMaster memory to something around 2G to 4G and see if it helps. The parameters can be found here: in ambari UI
"ams-env" --> "metrics_collector_heapsize" "ams-hbase-env" --> "hbase_master_heapsize"
For More accurate Tuning Recommendations (data needed)
For more accurate tuning of AMS service you may need to collect some additional details like the output fo the following API calls and the current memory usage:
1. The JSON response from the following API calls to see the uni queue metrics and hosts, to understand the load on AMS.
http://<ams-host>:6188/ws/v1/timeline/metrics/metadata http://<ams-host>:6188/ws/v1/timeline/metrics/hosts
2. Current memory usages (either complete GC logs HMaster /var/log/ambari-metrics-collector/gc.log-xxxxxx and for AMS collector GC log : /var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx )
3. The output of the AMS HMaster jmx call: AMS HBase jmx Snapshot -
http://<METRICS_COLLECTOR_HOST>:61310/jmx
4. The AMS HMaster UI output to understand the "Region Count" and "StoreFile Count"
http://<METRICS_COLLECTOR_HOST>:61310
5. AMS collector and HMaster GC complete logs.
6. AMS Configurations:
# tar czhfv /tmp/ams_collector_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ambari-metrics-collector/ # tar czhfv /tmp/ams_hmaster_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ams-hbase/
.
Created 08-29-2018 10:01 PM
You seems to be looking at the Ambari Dashboard.
Can you please click on the individual service like "HDFS" and then check the widgets in "Summary" section bottom.
Also every service does not provide the metrics via Widgets like Zookeeper/ SmartSense/Slider so you will not see metrics widgets for them.
Also you can check the metrics details using Grafana UI which actually pulls the metrics details from Ambari Metrics Collector.
Created on 08-29-2018 10:21 PM - edited 08-18-2019 02:48 AM
i check this:
->Ambari metrics dashboard. ->cat /var/log/ambari-metrics-collector/gc.log-201808291552 ->cat /var/log/ambari-metrics-collector/hbase-ams-master-maestro.hdp.com.log |grep -i pause
Created 08-29-2018 10:43 PM
Few things:
The screenshot "memory.png" is showing the GC log of HMaster process and not for the Ambari Metrics Collector process.
The GC log file for AMS will be this one "/var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx"
.
The "pause.png" screenshot shows some INFO messages and the JVMUtil pause messages are mostly concerned when they are WARN level. INFO messages about the pause can keep appearing in the logs. So you should do this to actually see the Pauses which needs to be reviewed.
# cat /var/log/ambari-metrics-collector/hbase-ams-master-xxxxxxxxx.log | grep -i pause | grep -i WARN
General Recommendation (trial basis)
Also if you are running AMS in Embedded Mode default settigns then i will suggest you to try increasing the AMS collector heap size from default 512 MB to a bit high value lik2 1G to 2G and then restart the AMS Collector again. Similarly try increasing the HMaster memory to something around 2G to 4G and see if it helps. The parameters can be found here: in ambari UI
"ams-env" --> "metrics_collector_heapsize" "ams-hbase-env" --> "hbase_master_heapsize"
For More accurate Tuning Recommendations (data needed)
For more accurate tuning of AMS service you may need to collect some additional details like the output fo the following API calls and the current memory usage:
1. The JSON response from the following API calls to see the uni queue metrics and hosts, to understand the load on AMS.
http://<ams-host>:6188/ws/v1/timeline/metrics/metadata http://<ams-host>:6188/ws/v1/timeline/metrics/hosts
2. Current memory usages (either complete GC logs HMaster /var/log/ambari-metrics-collector/gc.log-xxxxxx and for AMS collector GC log : /var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx )
3. The output of the AMS HMaster jmx call: AMS HBase jmx Snapshot -
http://<METRICS_COLLECTOR_HOST>:61310/jmx
4. The AMS HMaster UI output to understand the "Region Count" and "StoreFile Count"
http://<METRICS_COLLECTOR_HOST>:61310
5. AMS collector and HMaster GC complete logs.
6. AMS Configurations:
# tar czhfv /tmp/ams_collector_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ambari-metrics-collector/ # tar czhfv /tmp/ams_hmaster_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ams-hbase/
.
Created 08-29-2018 10:45 PM
There are some very useful Ambari Metrics Tuning Articles / docs available here which you might want to refer:
1. https://cwiki.apache.org/confluence/display/AMBARI/Configurations+-+Tuning
2. https://community.hortonworks.com/content/supportkb/208353/troubleshooting-ambari-metrics-ams.html
Created 08-30-2018 05:02 PM
can be also a ram memory problem?
Created 08-30-2018 11:21 PM
Yes, Insufficient RAM can be a problem. However this can be investigated/confirmed based on the data requested above.