Cloudera Community
avembu
user rank
Contributor
My Accepted Solutions
Show All

Node manager immediately goes down after starting ...

by Contributor in Support Questions
‎05-18-2019 03:29 PM
‎05-18-2019 03:29 PM
I am trying to start the HDFS service from Ambari. All the sub components comes up except the Nodemanager service under HDFS I am seeing the below error message in the /var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-<Node_Name>.log 2019-05-17 13:36:03,850 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - NodeManager metrics system stopped. 2019-05-17 13:36:03,850 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(606)) - NodeManager metrics system shutdown complete. 2019-05-17 13:36:03,851 FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager org.apache.hadoop.service.ServiceStateException: java.net.BindException: Address already in use at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.serviceInit(AuxiliaryServiceWithCustomClassLoader.java:65) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:162) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:546) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:594) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128) at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:504) at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1226) at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:495) at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:480) at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:213) at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:355) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at java.lang.Thread.run(Thread.java:748) 2019-05-17 13:36:03,853 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG: /************************************************************ Observations made No process running the nodemanager port - 0.0.0.0:45454 and 8042 Also no nodemanager process is running - ps -ef | grep nodemanager is returning empty result Any help would be appreciated ... View more
Labels:

Re: Zookeeper Failover Controller failed to start

by Contributor in Support Questions
‎03-07-2018 01:46 AM
‎03-07-2018 01:46 AM
Thanks @Geoffrey Shelton Okot The issue has been resolved. Again I came to know the importance of /etc/hosts file. It's not the firewall that was blocking the connection rather the process was spawned internal to the instance - meaning none of the other instance could access the process. Zookeeper process looks for the ip address from /etc/hosts file and spawns the process, instead of fetching the ip address it took the loopback address(127.0.0.1) which made sure the outside world cannot access the process. Followed the thread to resolve the issue MeaningOfIPaddressinProcess ... View more

Re: Zookeeper Failover Controller failed to start

by Contributor in Support Questions
‎03-07-2018 12:07 AM
‎03-07-2018 12:07 AM
@Geoffrey Shelton Okot We have disabled firewall already for all the hosts in the cluster. Also the port for which we are getting connection refused is the one which has the process running internal to the instance - meaning only localhost can access that process. Not sure why we are getting connection refused for a process that is running internal to an instance. Attached the screenshot where the process is internal to 127.0.1.1. Any inputs would be appreciated? ... View more

Re: Zookeeper Failover Controller failed to start

by Contributor in Support Questions
‎03-06-2018 11:10 PM
‎03-06-2018 11:10 PM
@Geoffrey Shelton Okot Thanks for the response. Zookeeper is running on these ports zookeeper-server1.pngzookeeper-server2.png. Attaching the process screenshots. I am not able to telnet to that port as well from the node where we are seeing the error like telnet host1/host2 3888. Can it be due to the fact that fire wall has been set? But I am able to telnet to the port 2181 - I thought 2181 is the default zookeeper port. Please confirm? ... View more

Zookeeper Failover Controller failed to start

by Contributor in Support Questions
‎03-06-2018 07:46 PM
‎03-06-2018 07:46 PM
I am trying to bring up a Hortonworks cluster. Below are the services in the cluster that I am trying to install Zookeeper Ambari metrics HDFS YARN MR2 Out the above services I was able to bring up the Zookeeper and Ambari metrics services. But the other services(HDFS, YARN and MR2) are not coming up. Namenode is also not coming up. I am trying to install the cluster in 3 nodes which is HA as well. When I checked the HDFS alerts one of the critical alert was that Zookeeper Failover Controller hasn't been started. After googling I tried to format it using the command hdfs zkfc -formatZK -nonInteractive but getting same error as I am getting the Ambari UI. My feeling is that ZKFC startup is causing the other hadoop services not to start. Below is the error message from the Zookeeper logs 2018-03-06 13:34:20,580 - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@383] - Cannot open channel to 3 at election address Host2/ip3-host4:3888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:404) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:840) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:795) Attaching the below items Zookeeper log from the namenode Ambari UI log I have been struck with this for the past 2 days. I tried uninstalling and reinstalling the cluster 2 times but still getting the same error. Any inputs would be appreciated. ... View more
Labels:

Re: When trying to install cluster via Ambari it's...

by Contributor in Support Questions
‎09-18-2017 07:25 PM
‎09-18-2017 07:25 PM
hi @Sonu Sahi : After reseting the Ambari I was able to get back the installation screen again. Thanks ... View more

When trying to install cluster via Ambari it's sto...

by Contributor in Support Questions
‎09-15-2017 03:54 PM
‎09-15-2017 03:54 PM
ambari-step9-blank.jpg ... View more
Labels:

Re: unable to start postgres for ambari

by Contributor in Support Questions
‎04-06-2017 05:29 PM
‎04-06-2017 05:29 PM
oh okay. any clue on what the error might be? ... View more

Re: unable to start postgres for ambari

by Contributor in Support Questions
‎04-06-2017 04:36 PM
‎04-06-2017 04:36 PM
hi @Sunile Manjee: Were you able to resolve the issue? ... View more
Contact Me
Online
Offline
Last Visited
‎05-20-2019 04:08 PM
Community Statistics
Member Since ‎04-06-2017 04:34 PM
Last Visited ‎05-20-2019 04:08 PM
Posts 17
Kudos received 1