Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

why i can't see the ambari-metrics, my ambari metrics service is installed

Solved Go to solution
Highlighted

why i can't see the ambari-metrics, my ambari metrics service is installed

Explorer

88389-am-me-no-ava.png

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

Super Mentor

@yadir Aguilar

Few things:
The screenshot "memory.png" is showing the GC log of HMaster process and not for the Ambari Metrics Collector process.
The GC log file for AMS will be this one "/var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx"
.

The "pause.png" screenshot shows some INFO messages and the JVMUtil pause messages are mostly concerned when they are WARN level. INFO messages about the pause can keep appearing in the logs. So you should do this to actually see the Pauses which needs to be reviewed.

# cat /var/log/ambari-metrics-collector/hbase-ams-master-xxxxxxxxx.log  | grep -i pause | grep -i WARN


General Recommendation (trial basis)
Also if you are running AMS in Embedded Mode default settigns then i will suggest you to try increasing the AMS collector heap size from default 512 MB to a bit high value lik2 1G to 2G and then restart the AMS Collector again. Similarly try increasing the HMaster memory to something around 2G to 4G and see if it helps. The parameters can be found here: in ambari UI

"ams-env" --> "metrics_collector_heapsize" 
"ams-hbase-env" --> "hbase_master_heapsize"



For More accurate Tuning Recommendations (data needed)
For more accurate tuning of AMS service you may need to collect some additional details like the output fo the following API calls and the current memory usage:

1. The JSON response from the following API calls to see the uni queue metrics and hosts, to understand the load on AMS.

http://<ams-host>:6188/ws/v1/timeline/metrics/metadata
http://<ams-host>:6188/ws/v1/timeline/metrics/hosts 


2. Current memory usages (either complete GC logs HMaster /var/log/ambari-metrics-collector/gc.log-xxxxxx and for AMS collector GC log : /var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx )

3. The output of the AMS HMaster jmx call: AMS HBase jmx Snapshot -

http://<METRICS_COLLECTOR_HOST>:61310/jmx


4. The AMS HMaster UI output to understand the "Region Count" and "StoreFile Count"

http://<METRICS_COLLECTOR_HOST>:61310


5. AMS collector and HMaster GC complete logs.

6. AMS Configurations:

# tar czhfv /tmp/ams_collector_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ambari-metrics-collector/ 
# tar czhfv /tmp/ams_hmaster_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ams-hbase/ 

.

View solution in original post

6 REPLIES 6
Highlighted

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

Super Mentor

@yadir Aguilar

You seems to be looking at the Ambari Dashboard.

Can you please click on the individual service like "HDFS" and then check the widgets in "Summary" section bottom.

Also every service does not provide the metrics via Widgets like Zookeeper/ SmartSense/Slider so you will not see metrics widgets for them.

Also you can check the metrics details using Grafana UI which actually pulls the metrics details from Ambari Metrics Collector.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-operations/content/grafana_dashbo...

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

Explorer

@Jay Kumar SenSharma

i check this:

->Ambari metrics dashboard. 

->cat /var/log/ambari-metrics-collector/gc.log-201808291552

->cat /var/log/ambari-metrics-collector/hbase-ams-master-maestro.hdp.com.log |grep -i pause

88390-am-me-no-ava.png

88391-pause.png

88392-memory.png

Highlighted

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

Super Mentor

@yadir Aguilar

Few things:
The screenshot "memory.png" is showing the GC log of HMaster process and not for the Ambari Metrics Collector process.
The GC log file for AMS will be this one "/var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx"
.

The "pause.png" screenshot shows some INFO messages and the JVMUtil pause messages are mostly concerned when they are WARN level. INFO messages about the pause can keep appearing in the logs. So you should do this to actually see the Pauses which needs to be reviewed.

# cat /var/log/ambari-metrics-collector/hbase-ams-master-xxxxxxxxx.log  | grep -i pause | grep -i WARN


General Recommendation (trial basis)
Also if you are running AMS in Embedded Mode default settigns then i will suggest you to try increasing the AMS collector heap size from default 512 MB to a bit high value lik2 1G to 2G and then restart the AMS Collector again. Similarly try increasing the HMaster memory to something around 2G to 4G and see if it helps. The parameters can be found here: in ambari UI

"ams-env" --> "metrics_collector_heapsize" 
"ams-hbase-env" --> "hbase_master_heapsize"



For More accurate Tuning Recommendations (data needed)
For more accurate tuning of AMS service you may need to collect some additional details like the output fo the following API calls and the current memory usage:

1. The JSON response from the following API calls to see the uni queue metrics and hosts, to understand the load on AMS.

http://<ams-host>:6188/ws/v1/timeline/metrics/metadata
http://<ams-host>:6188/ws/v1/timeline/metrics/hosts 


2. Current memory usages (either complete GC logs HMaster /var/log/ambari-metrics-collector/gc.log-xxxxxx and for AMS collector GC log : /var/log/ambari-metrics-collector/collector-gc.logxxxxxxxx )

3. The output of the AMS HMaster jmx call: AMS HBase jmx Snapshot -

http://<METRICS_COLLECTOR_HOST>:61310/jmx


4. The AMS HMaster UI output to understand the "Region Count" and "StoreFile Count"

http://<METRICS_COLLECTOR_HOST>:61310


5. AMS collector and HMaster GC complete logs.

6. AMS Configurations:

# tar czhfv /tmp/ams_collector_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ambari-metrics-collector/ 
# tar czhfv /tmp/ams_hmaster_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ams-hbase/ 

.

View solution in original post

Highlighted

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

Super Mentor
Highlighted

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

Explorer

@Jay Kumar SenSharma

can be also a ram memory problem?

Highlighted

Re: why i can't see the ambari-metrics, my ambari metrics service is installed

Super Mentor

Yes, Insufficient RAM can be a problem. However this can be investigated/confirmed based on the data requested above.

Don't have an account?
Coming from Hortonworks? Activate your account here