Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

why i can't see the ambari-metrics, my ambari metrics service is installed

avatar
Explorer

88389-am-me-no-ava.png

1 ACCEPTED SOLUTION

avatar
Master Mentor

@yadir Aguilar

Few things:
The screenshot "memory.png" is showing the GC log of HMaster process and not for the Ambari Metrics Collector process.
The GC log file for AMS will be this one "/var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx"
.

The "pause.png" screenshot shows some INFO messages and the JVMUtil pause messages are mostly concerned when they are WARN level. INFO messages about the pause can keep appearing in the logs. So you should do this to actually see the Pauses which needs to be reviewed.

# cat /var/log/ambari-metrics-collector/hbase-ams-master-xxxxxxxxx.log  | grep -i pause | grep -i WARN


General Recommendation (trial basis)
Also if you are running AMS in Embedded Mode default settigns then i will suggest you to try increasing the AMS collector heap size from default 512 MB to a bit high value lik2 1G to 2G and then restart the AMS Collector again. Similarly try increasing the HMaster memory to something around 2G to 4G and see if it helps. The parameters can be found here: in ambari UI

"ams-env" --> "metrics_collector_heapsize" 
"ams-hbase-env" --> "hbase_master_heapsize"



For More accurate Tuning Recommendations (data needed)
For more accurate tuning of AMS service you may need to collect some additional details like the output fo the following API calls and the current memory usage:

1. The JSON response from the following API calls to see the uni queue metrics and hosts, to understand the load on AMS.

http://<ams-host>:6188/ws/v1/timeline/metrics/metadata
http://<ams-host>:6188/ws/v1/timeline/metrics/hosts 


2. Current memory usages (either complete GC logs HMaster /var/log/ambari-metrics-collector/gc.log-xxxxxx and for AMS collector GC log : /var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx )

3. The output of the AMS HMaster jmx call: AMS HBase jmx Snapshot -

http://<METRICS_COLLECTOR_HOST>:61310/jmx


4. The AMS HMaster UI output to understand the "Region Count" and "StoreFile Count"

http://<METRICS_COLLECTOR_HOST>:61310


5. AMS collector and HMaster GC complete logs.

6. AMS Configurations:

# tar czhfv /tmp/ams_collector_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ambari-metrics-collector/ 
# tar czhfv /tmp/ams_hmaster_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ams-hbase/ 

.

View solution in original post

6 REPLIES 6

avatar
Master Mentor

@yadir Aguilar

You seems to be looking at the Ambari Dashboard.

Can you please click on the individual service like "HDFS" and then check the widgets in "Summary" section bottom.

Also every service does not provide the metrics via Widgets like Zookeeper/ SmartSense/Slider so you will not see metrics widgets for them.

Also you can check the metrics details using Grafana UI which actually pulls the metrics details from Ambari Metrics Collector.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/grafana_dashbo...

avatar
Explorer

@Jay Kumar SenSharma

i check this:

->Ambari metrics dashboard. 

->cat /var/log/ambari-metrics-collector/gc.log-201808291552

->cat /var/log/ambari-metrics-collector/hbase-ams-master-maestro.hdp.com.log |grep -i pause

88390-am-me-no-ava.png

88391-pause.png

88392-memory.png

avatar
Master Mentor

@yadir Aguilar

Few things:
The screenshot "memory.png" is showing the GC log of HMaster process and not for the Ambari Metrics Collector process.
The GC log file for AMS will be this one "/var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx"
.

The "pause.png" screenshot shows some INFO messages and the JVMUtil pause messages are mostly concerned when they are WARN level. INFO messages about the pause can keep appearing in the logs. So you should do this to actually see the Pauses which needs to be reviewed.

# cat /var/log/ambari-metrics-collector/hbase-ams-master-xxxxxxxxx.log  | grep -i pause | grep -i WARN


General Recommendation (trial basis)
Also if you are running AMS in Embedded Mode default settigns then i will suggest you to try increasing the AMS collector heap size from default 512 MB to a bit high value lik2 1G to 2G and then restart the AMS Collector again. Similarly try increasing the HMaster memory to something around 2G to 4G and see if it helps. The parameters can be found here: in ambari UI

"ams-env" --> "metrics_collector_heapsize" 
"ams-hbase-env" --> "hbase_master_heapsize"



For More accurate Tuning Recommendations (data needed)
For more accurate tuning of AMS service you may need to collect some additional details like the output fo the following API calls and the current memory usage:

1. The JSON response from the following API calls to see the uni queue metrics and hosts, to understand the load on AMS.

http://<ams-host>:6188/ws/v1/timeline/metrics/metadata
http://<ams-host>:6188/ws/v1/timeline/metrics/hosts 


2. Current memory usages (either complete GC logs HMaster /var/log/ambari-metrics-collector/gc.log-xxxxxx and for AMS collector GC log : /var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx )

3. The output of the AMS HMaster jmx call: AMS HBase jmx Snapshot -

http://<METRICS_COLLECTOR_HOST>:61310/jmx


4. The AMS HMaster UI output to understand the "Region Count" and "StoreFile Count"

http://<METRICS_COLLECTOR_HOST>:61310


5. AMS collector and HMaster GC complete logs.

6. AMS Configurations:

# tar czhfv /tmp/ams_collector_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ambari-metrics-collector/ 
# tar czhfv /tmp/ams_hmaster_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ams-hbase/ 

.

avatar
Master Mentor

avatar
Explorer

@Jay Kumar SenSharma

can be also a ram memory problem?

avatar
Master Mentor

Yes, Insufficient RAM can be a problem. However this can be investigated/confirmed based on the data requested above.