ambari metrics collector connection refused : metrics data not available on Ambari Dashboard

[#1]

amabri metrics collector is starting again and error is : connection refused, I have gone through logs and found some errors:

because of this there are no metrics available on Ambari Dashboard

ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: HBaseAccessor getConnection failed after 10 attempts

ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Error creating Metrics Schema in HBase using Phoenix.

FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer

org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Error creating Metrics Schema in HBase using Phoenix.

I have increased values for some properties :

metrics_collector_heap_size, hbase_regionserver_heapsize, hbase_master_heapsize, hbase_master_xmn_size

but still Metrics is not available

[#2]

@Anurag Mishra

can you please confirm which version of Ambari you are using and also please confirm whether you are using the Ambari Metrics in EMBEDDED or DISTRIBUTED MODE

Can you please check your YARN APP Timeline server is running and it is not having any issues.

you can refer this for more details:

https://community.hortonworks.com/questions/73577/problem-with-ambari-metrics.html

Hope this helps.

[#3]

Ambari : Version 2.4.1.0

MODE: embedded

[#4]

@Anurag Mishra

You might find some more ERROR logging prior to the line which might be helpful to look:

ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor:
 HBaseAccessor getConnection failed after 10 attempts

.

Can you please share that complete log.

What is the Cluster Size (Number of Hosts and what all Services are installed to it)

.

- Also can you please try cleaning up the following

- Specially the "hbase.zookeeper.property.dataDir" -> Remove the AMS zookeeper data by backing up and removing the contents of 'hbase.tmp.dir'/zookeeper

- Remove any Phoenix spool files from 'hbase.tmp.dir'/phoenix-spool folder

- The try restarting the AMS

[#5]

12 hosts with 503 RAM each, services : HDFS,YARN,MapReduce2,Tez,Hive,HBase,Pig,Sqoop,Oozie,ZooKeeper,Ambari Infra,Ambari Metrics,Atlas,Kafka,Spark,Spark2,Zeppelin

[#6]

Thank you very much! It helped me to solve my problem with AMS.

[#7]

@Rustem Iskakov

Great !!!