question Re: Ambari Metrics Collector Start Failed on 3 Node AWS Cluster in Support Questions

Ambari Metrics Collector Start Failed on 3 Node AWS Cluster

rainer_geissend — Mon, 19 Aug 2019 11:03:44 GMT

I am on Ambari Version 2.2.0.0 and did a fresh install of hadoop.

Whenever I start my hadoop services I get the following warnings and my metrics collector is shutting down (all other hadoop services are fine):

I attached all the logs from the ambari-metrics-collector directory in the attached zip.

Any help is appreciated!

ambari-metrics-collector.zip

br,

Rainer

Re: Ambari Metrics Collector Start Failed on 3 Node AWS Cluster

amcbarnett — Thu, 28 Jan 2016 03:03:36 GMT

I haven't looked at you log files as yet but see the following to solve common issues.

Up the heap size ams-env : metrics_collector_heapsize = 1024
Set timeline.metrics.service.default.result.limit = 15840 
Restart the Collector

https://cwiki.apache.org/confluence/display/AMBARI/Configurations+-+Tuning

https://community.hortonworks.com/articles/11805/how-to-solve-ambari-metrics-corrupted-data.html

https://community.hortonworks.com/questions/8928/ambari-metrics-1.html

Re: Ambari Metrics Collector Start Failed on 3 Node AWS Cluster

jstraub — Thu, 28 Jan 2016 03:20:07 GMT

It looks like your Metrics Collector cannot start because the HBase Master is not coming up. Just to make sure you're running a non-kerberized environment with a single Namenode, right? Are you using Metrics in distributed or embedded mode?

Could you please validate and post this configuration => hbase.rootdir

Your HBase Master log files show a connection refused when the HBase Master is trying to connect to the Namenode

2016-01-27 18:36:57,189 FATAL [hdp1n3:61300.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call From hdp1n3/XXXXXXXXX to hdp1n3.aye1vpcdev:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
	at org.apache.hadoop.ipc.Client.call(Client.java:1431)
	...
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	...

Is this the hostname of your namenode "hdp1n3.aye1vpcdev:8020"?

Can you access the HDFS from the Metrics Collector node?

Re: Ambari Metrics Collector Start Failed on 3 Node AWS Cluster

nsabharwal — Thu, 28 Jan 2016 04:32:03 GMT

@Ancil McBarnett This has worked for me in the past.

Re: Ambari Metrics Collector Start Failed on 3 Node AWS Cluster

rainer_geissend — Fri, 29 Jan 2016 00:04:03 GMT

Thanks Ancil, I tried this already and unfortunately it didn't work out for me ...

Re: Ambari Metrics Collector Start Failed on 3 Node AWS Cluster

rainer_geissend — Fri, 29 Jan 2016 00:19:01 GMT

Hi Jonas, thanks, this was exactly the issue ... I switched from Embedded to Distributed Mode and by doing that, the default value for hbase.rootdir was pointing to the wrong note ... now everything is working as expected! Thanks again for your feedback, highly appreciated!

Re: Ambari Metrics Collector Start Failed on 3 Node AWS Cluster

jstraub — Fri, 29 Jan 2016 02:35:56 GMT

Glad it worked! Happy Hadooping 🙂

Re: Ambari Metrics Collector Start Failed on 3 Node AWS Cluster

nsabharwal — Fri, 29 Jan 2016 09:54:56 GMT

@Jonas Straub Happy Hadooping!! 🙂