Member since
02-23-2018
3
Posts
0
Kudos Received
0
Solutions
03-01-2018
08:16 AM
@Geoffrey Shelton Okot Thank you for comments. It worked after iptables rules clear.
... View more
02-28-2018
11:57 AM
Some update on this issue........ It Seems to Zookeeper server is not running properly. While checking server status in zookeeper, following error message is displayed [root@jn2 bin]# ./zkServer.sh status ZooKeeper JMX enabled by default Using config: /opt/software/zookeeper/bin/../conf/zoo.cfg Error contacting service. It is probably not running. ZooKeeper Logs :: 2018-02-28 06:46:11,718 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@167] - Resolved hostname: jn3 to address: jn3/15.34.71.187 2018-02-28 06:46:11,718 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@852] - Notification time out: 60000 2018-02-28 06:47:11,723 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 2 at election address jn2/15.34.71.84:3888 java.net.NoRouteToHostException: No route to host (Host unreachable) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:614) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:913) 2018-02-28 06:47:11,723 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@167] - Resolved hostname: jn2 to address: jn2/15.34.71.84 2018-02-28 06:47:11,724 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 3 at election address jn3/15.34.71.187:3888 java.net.NoRouteToHostException: No route to host (Host unreachable) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:614) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:913) 2018-02-28 06:47:11,725 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@167] - Resolved hostname: jn3 to address: jn3/15.34.71.187 2018-02-28 06:47:11,725 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@852] - Notification time out: 60000 ################################################################# but in jps, I can observe proper daemon services are running. 2176 QuorumPeerMain 2257 JournalNode 2458 NodeManager 2349 DataNode 2606 Jps Not sure which configuration issue causes this. Looking forward to get some inputs from this community.
... View more
02-23-2018
04:34 PM
Hi, Am trying to learn Hadoop by self learning. I've created 8 instance by using VMWare Workstation. 2 - name nodes (i.e. nn1,nn2) ,3 data nodes(dn1,dn2,dn3),3 journal nodes(jn1,jn2,jn3). During start of each node, expect Name Nodes, rest all started successfully. Name node is getting shutdown because below reasons as per my understanding. 1) Issue in hadoop-root-zkfc-nn1.log
2018-02-23 06:41:28,449 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=jn1:2181,jn2:2181,jn3:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@16d04d3d 2018-02-23 06:41:28,706 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 15.34.71.241/15.34.71.241:2181. Will not attempt to authenticate using SASL (unknown error) 2018-02-23 06:41:28,713 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2018-02-23 06:41:28,889 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 15.34.71.84/15.34.71.84:2181. Will not attempt to authenticate using SASL (unknown error) 2018-02-23 06:41:28,891 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect 2) Issue in hadoop-root-namenode-nn1.log
2018-02-22 11:47:57,913 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /opt/software/hdfsdrive/name is in an inconsistent state: storage directory does not exist or is not accessible. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:374) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:225) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:978) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:685) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:819) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:803) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566) 2018-02-22 11:47:57,915 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 2018-02-22 11:47:57,948 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: It seems to Because of issue 1, issue 2 also observed. I could able to login from each device to another device by ssh. verified hostname in each device which is correct. zoo.cfg also configured properly.
dataDir=/opt/software/zookeeper/data dataLogDir=/opt/software/zookeeper/logs server.1=jn1:2888:3888 server.2=jn2:2889:3889 server.3=jn3:2890:3890 status of zookeeper in jn1: [root@jn1 ~]# jps 1376 QuorumPeerMain 1449 JournalNode 1530 DataNode 1626 NodeManager 1820 Jps Can any one help to resolve this issue. Thanks in advance
... View more
Labels:
- Labels:
-
Apache Hadoop