Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

Not able to connect Zookeeper server from Client...

Labels:
avatar
author-rank pratti4u
New Contributor

Created ‎02-23-2018 04:34 PM

Hi,

Am trying to learn Hadoop by self learning.

I've created 8 instance by using VMWare Workstation.

2 - name nodes (i.e. nn1,nn2) ,3 data nodes(dn1,dn2,dn3),3 journal nodes(jn1,jn2,jn3).

During start of each node, expect Name Nodes, rest all started successfully.

Name node is getting shutdown because below reasons as per my understanding.

1) Issue in hadoop-root-zkfc-nn1.log

2018-02-23 06:41:28,449 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=jn1:2181,jn2:2181,jn3:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@16d04d3d

2018-02-23 06:41:28,706 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 15.34.71.241/15.34.71.241:2181. Will not attempt to authenticate using SASL (unknown error)

2018-02-23 06:41:28,713 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.NoRouteToHostException: No route to host

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)

at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

2018-02-23 06:41:28,889 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server 15.34.71.84/15.34.71.84:2181. Will not attempt to authenticate using SASL (unknown error)

2018-02-23 06:41:28,891 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

2) Issue in hadoop-root-namenode-nn1.log

2018-02-22 11:47:57,913 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /opt/software/hdfsdrive/name is in an inconsistent state: storage directory does not exist or is not accessible.

at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:374)

at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:225)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:978)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:685)

at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:819)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:803)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1500)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1566)

2018-02-22 11:47:57,915 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1

2018-02-22 11:47:57,948 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

It seems to Because of issue 1, issue 2 also observed.

I could able to login from each device to another device by ssh.

verified hostname in each device which is correct.

zoo.cfg also configured properly.

dataDir=/opt/software/zookeeper/data

dataLogDir=/opt/software/zookeeper/logs

server.1=jn1:2888:3888

server.2=jn2:2889:3889

server.3=jn3:2890:3890

status of zookeeper in jn1:


[root@jn1 ~]# jps
1376 QuorumPeerMain
1449 JournalNode
1530 DataNode
1626 NodeManager
1820 Jps

Can any one help to resolve this issue.

Thanks in advance

344,855 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank Shelton
Master Mentor

Created ‎02-28-2018 02:10 PM

@Ravikanth Pratti

The below error is typical of firewall issue.Make sure firewall is not blocking your access iptables is default firewall on Linux. Run following command to see what iptables rules are setup: 

# /sbin/iptables -L -n

Firewall error

0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 3 at election address jn3/15.34.71.187:3888 java.net.NoRouteToHostException: No route to host (Host unreachable)

You can temporary clear all iptables rules so that you can troubleshoot problem. If you are using Red Hat or Fedora Linux type command: 

# /etc/init.d/iptables save 
# /etc/init.d/iptables

stop If you are using other Linux distribution type following commands: 

# iptables -F 
# iptables -X 
# iptables -t nat -F 
# iptables -t nat -X 
# iptables -t mangle -F 
# iptables -t mangle -X

Hope that helps

View solution in original post

342,873 Views
3 REPLIES 3

avatar
author-rank pratti4u
New Contributor

Created ‎02-28-2018 11:57 AM

Some update on this issue........

It Seems to Zookeeper server is not running properly.

While checking server status in zookeeper, following error message is displayed

[root@jn2 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/software/zookeeper/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.

ZooKeeper Logs ::

2018-02-28 06:46:11,718 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@167] - Resolved hostname: jn3 to address: jn3/15.34.71.187

2018-02-28 06:46:11,718 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@852] - Notification time out: 60000

2018-02-28 06:47:11,723 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 2 at election address jn2/15.34.71.84:3888

java.net.NoRouteToHostException: No route to host (Host unreachable)

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:614)

at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)

at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:913)

2018-02-28 06:47:11,723 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@167] - Resolved hostname: jn2 to address: jn2/15.34.71.84

2018-02-28 06:47:11,724 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 3 at election address jn3/15.34.71.187:3888

java.net.NoRouteToHostException: No route to host (Host unreachable)

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562)

at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:614)

at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)

at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:913)

2018-02-28 06:47:11,725 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@167] - Resolved hostname: jn3 to address: jn3/15.34.71.187

2018-02-28 06:47:11,725 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@852] - Notification time out: 60000

#################################################################

but in jps, I can observe proper daemon services are running.
2176 QuorumPeerMain
2257 JournalNode
2458 NodeManager
2349 DataNode
2606 Jps

Not sure which configuration issue causes this. Looking forward to get some inputs from this community.

342,873 Views
0 Kudos

avatar
author-rank Shelton
Master Mentor

Created ‎02-28-2018 02:10 PM

@Ravikanth Pratti

The below error is typical of firewall issue.Make sure firewall is not blocking your access iptables is default firewall on Linux. Run following command to see what iptables rules are setup: 

# /sbin/iptables -L -n

Firewall error

0:0:0:2181:QuorumCnxManager@588] - Cannot open channel to 3 at election address jn3/15.34.71.187:3888 java.net.NoRouteToHostException: No route to host (Host unreachable)

You can temporary clear all iptables rules so that you can troubleshoot problem. If you are using Red Hat or Fedora Linux type command: 

# /etc/init.d/iptables save 
# /etc/init.d/iptables

stop If you are using other Linux distribution type following commands: 

# iptables -F 
# iptables -X 
# iptables -t nat -F 
# iptables -t nat -X 
# iptables -t mangle -F 
# iptables -t mangle -X

Hope that helps

342,874 Views

avatar
author-rank pratti4u
New Contributor

Created ‎03-01-2018 08:16 AM

@Geoffrey Shelton Okot

Thank you for comments.

It worked after iptables rules clear.

342,873 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements