cant start the zookeeper server on masters machines

[#1]

we have 3 masters machine in ambari cluster

the first service that need to start is the zookeper server on all masters ( master01/02/03 )

but zookeeper service not start on the master01 and master02 machines

from /var/log/zookeper we see the following :

what chould be the problem ?

2017-12-03 19:54:29,832 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /120.14.51.19:59808
2017-12-03 19:54:29,838 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /120.14.51.19:59808
2017-12-03 19:54:29,844 - INFO  [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x1601dccc7e8049d with negotiated timeout 30000 for client /120.14.51.19:59808
2017-12-03 19:54:31,394 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1601dccc7e8049d, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2017-12-03 19:54:31,395 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /120.14.51.19:59808 which had sessionid 0x1601dccc7e8049d
2017-12-03 19:54:33,474 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /120.14.51.19:59810
2017-12-03 19:54:33,482 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /120.14.51.19:59810
2017-12-03 19:54:33,492 - INFO  [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x1601dccc7e8049e with negotiated timeout 30000 for client /120.14.51.19:59810
2017-12-03 19:54:35,150 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1601dccc7e8049e, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2017-12-03 19:54:35,151 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /120.14.51.19:59810 which had sessionid 0x1601dccc7e8049e
2017-12-03 19:54:35,378 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /120.14.51.17:50721
2017-12-03 19:54:35,388 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /120.14.51.17:50721
2017-12-03 19:54:35,397 - INFO  [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x1601dccc7e8049f with negotiated timeout 30000 for client /120.14.51.17:50721
^C

[#2]

Hello Michael,

These messages doesn't seem to be the root cause of the issue, as the error mentions, they are just some clients releasing their stream from the zookeeper.

• Verify if it has sufficient space on all system, log, and hdfs partitions
• Some heapsize configuration issues ?
• Does it have the correct permissions within its directory ? ( Including owner / group )
• Have you tried to check the ".out" files generated by zookeeper ?

Regards,

[#3]

from the out file we get that --> Getting jmx metrics from NN failed. URL: http://master03.sys56.com:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem