Support Questions
Find answers, ask questions, and share your expertise

ambari metrics not starting

Explorer

Hi All,

I am new to hortonworks and facing some issues while starting the Ambari metrics and below is the content of ambari-metrics-collector.log. Could anyone help me on this?

2017-08-04 15:30:36,943 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2017-08-04 15:30:36,965 INFO org.apache.hadoop.http.HttpServer2: HttpServer.start() threw a non Bind IOException java.net.BindException: Port in use: server1.cloudera.com:6188 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:919) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856) at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:347) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.startWebApp(ApplicationHistoryServer.java:180) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceStart(ApplicationHistoryServer.java:92) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:138) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:147) Caused by: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:914) ... 7 more 2017-08-04 15:30:36,967 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: AHSWebApp failed to start. org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:351) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.startWebApp(ApplicationHistoryServer.java:180) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceStart(ApplicationHistoryServer.java:92) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:138) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:147) Caused by: java.net.BindException: Port in use: server1.cloudera.com:6188 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:919) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856) at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:347) ... 5 more Caused by: java.net.BindException: Cannot assign requested address at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:914) ... 7 more

Thanks,

Vijay

4 REPLIES 4

Super Mentor

@vijaybaskar lakshminarayanan

The error indicates a port conflict on port 6188

ava.net.BindException: Port in use: server1.cloudera.com:6188 at 

.

AMS uses 6188 port. So before starting AMS please check if any other process is consuming that port?

# netstat -tnlpa | grep 6188

.

Ambari UI --> Ambari Metrics --> "Configs" (tab) --> Advanced (child tab) --> navigate to "Advanced ams-site" and search for the following property:

timeline.metrics.service.webapp.address = 0.0.0.0:6188


You can change that port to something else to avoid port conflict. then try restarting AMS.

.

Explorer

Thanks jay, I have checked and no processes is using 6188.

Super Mentor

@vijaybaskar lakshminarayanan

Then it is possible that AMS "Auto Start Service" is ON for AMS so when you are trying to start the AMS, ... during that time the ambari-agent was already trying to start the AMS process due to auto start feature. that might have lead to the port conflict issue.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-operations/content/ch07s04.html (Added in Ambari 2.5)

- Older verison of ambari also has this feature for ambari metrics collector which is usuually defined inside the ambari.properties.


In older version of ambari (2.4 or earlier) the auto start was being controlled using the /etc/ambari-server/conf/ambari.properites as following:

recovery.enabled_components=METRICS_COLLECTOR 
recovery.type=AUTO_START

.

Please turn that off for testing and then see if you are still getting port conflict?

Explorer

Jay - I have updated the ambari.properties with the following settings and again getting the same error.

recovery.type=DEFAULT

recovery.enabled_components=METRICS_COLLECTOR

recovery.lifetime_max_count=1024

Thanks