Not able to connect Zookeeper server from Client...

[#1]

Hi,

Am trying to learn Hadoop by self learning.

I've created 8 instance by using VMWare Workstation.

2 - name nodes (i.e. nn1,nn2) ,3 data nodes(dn1,dn2,dn3),3 journal nodes(jn1,jn2,jn3).

During start of each node, expect Name Nodes, rest all started successfully.

Name node is getting shutdown because below reasons as per my understanding.

1) Issue in hadoop-root-zkfc-nn1.log

2018-02-23 06:41:28,449 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=jn1:2181,jn2:2181,jn3:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@16d04d3d

2018-02-23 06:41:28,706 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 15.34.71.241/15.34.71.241:2181. Will not attempt to authenticate using SASL (unknown error)

2018-02-23 06:41:28,713 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.NoRouteToHostException: No route to host

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

2018-02-23 06:41:28,889 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 15.34.71.84/15.34.71.84:2181. Will not attempt to authenticate using SASL (unknown error)

2018-02-23 06:41:28,891 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

2) Issue in hadoop-root-namenode-nn1.log

2018-02-22 11:47:57,913 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /opt/software/hdfsdrive/name is in an inconsistent state: storage directory does not exist or is not accessible.

at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:374)

at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:225)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:978)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:685)

at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:819)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:803)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566)

2018-02-22 11:47:57,915 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1

2018-02-22 11:47:57,948 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

It seems to Because of issue 1, issue 2 also observed.

I could able to login from each device to another device by ssh.

verified hostname in each device which is correct.

zoo.cfg also configured properly.

dataDir=/opt/software/zookeeper/data

dataLogDir=/opt/software/zookeeper/logs

server.1=jn1:2888:3888

server.2=jn2:2889:3889

server.3=jn3:2890:3890

status of zookeeper in jn1:

[root@jn1 ~]# jps
1376 QuorumPeerMain
1449 JournalNode
1530 DataNode
1626 NodeManager
1820 Jps

Can any one help to resolve this issue.

Thanks in advance

[#2]

Some update on this issue........

It Seems to Zookeeper server is not running properly.

While checking server status in zookeeper, following error message is displayed

[root@jn2 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.

ZooKeeper Logs ::

2018-02-28 06:46:11,718 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@167] - Resolved hostname: jn3 to address: jn3/15.34.71.187

2018-02-28 06:46:11,718 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@852] - Notification time out: 60000

2018-02-28 06:47:11,723 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 2 at election address jn2/15.34.71.84:3888

java.net.NoRouteToHostException: No route to host (Host unreachable)

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:614)

at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)

at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:913)

2018-02-28 06:47:11,723 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@167] - Resolved hostname: jn2 to address: jn2/15.34.71.84

2018-02-28 06:47:11,724 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 3 at election address jn3/15.34.71.187:3888

java.net.NoRouteToHostException: No route to host (Host unreachable)

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:614)

at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)

at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:913)

2018-02-28 06:47:11,725 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@167] - Resolved hostname: jn3 to address: jn3/15.34.71.187

2018-02-28 06:47:11,725 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@852] - Notification time out: 60000

#################################################################

but in jps, I can observe proper daemon services are running.
2176 QuorumPeerMain
2257 JournalNode
2458 NodeManager
2349 DataNode
2606 Jps

Not sure which configuration issue causes this. Looking forward to get some inputs from this community.

[#3 · Accepted solution]

@Ravikanth Pratti

The below error is typical of firewall issue.Make sure firewall is not blocking your access iptables is default firewall on Linux. Run following command to see what iptables rules are setup:

# /sbin/iptables -L -n 

Firewall error

0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 3 at election address jn3/15.34.71.187:3888 java.net.NoRouteToHostException: No route to host (Host unreachable) 

You can temporary clear all iptables rules so that you can troubleshoot problem. If you are using Red Hat or Fedora Linux type command:

# /etc/init.d/iptables save 
# /etc/init.d/iptables 

stop If you are using other Linux distribution type following commands:

# iptables -F 
# iptables -X 
# iptables -t nat -F 
# iptables -t nat -X 
# iptables -t mangle -F 
# iptables -t mangle -X

Hope that helps

[#4]

@Geoffrey Shelton Okot

Thank you for comments.

It worked after iptables rules clear.