Hi There,
The linux admins changed the root password of the hadoop box and then after 2 days reverted back. After that we have been having issues with the entire node. The disk wait and I/operations are taking longer than usual. I suspect issue with the zookeeper.
session established for client)
2019-08-27 22:23:31,908 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 1180ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2019-08-27 22:23:42,789 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 2060ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2019-08-27 22:24:01,821 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 1090ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2019-08-27 22:24:11,818 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 1086ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2019-08-27 22:24:20,787 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /host2:52159
2019-08-27 22:24:20,789 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /host2:52159
2019-08-27 22:24:20,789 - INFO [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x16cd24bcb850012 with negotiated timeout 180000 for client /host2:52159
2019-08-27 22:24:21,244 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /host2:52159 which had sessionid 0x16cd24bcb850012
2019-08-27 22:24:23,179 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 2446ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2019-08-27 22:24:30,245 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /host1:60574
2019-08-27 22:24:33,130 - WARN [SyncThread:1:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:1 took 2397ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
2019-08-27 22:24:34,710 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2019-08-27 22:24:34,710 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /host1:60574 (no session established for client)
