Member since
07-05-2016
14
Posts
0
Kudos Received
0
Solutions
11-16-2016
08:37 AM
@Jayanta Das Thanks for the update.
... View more
08-05-2016
02:58 PM
@sbhat Thanks for you reply. My issue falls under scenario 2 in that article. But my hive.pid permission are correct i.e -rw-r--r-- 1 hive hadoop 6 Aug 4 19:48 hive.pid
-rw-r--r-- 1 hive hadoop 6 Aug 4 19:48 hive-server.pid
And also my procees id's i.e hive-server.pid is matching with the process id when checked using ps -ef|grep hive
... View more
08-05-2016
02:49 PM
@Kuldeep Kulkarni Thanks for your reply. Restarted ambari agent, but it doesn't worked out.
... View more
08-04-2016
03:59 PM
Ambari dashboard is showing HiveServer2 and Hive Metastore is stopped, but from the command line I am able to login into hive cli and hiveserver2 using beeline client also. I have 6 node cluster in which I have installed hive server in 2 nodes and hive client is installed in all 6 nodes. In node1 ambari dashboard it was showing hiveserver2 and hive metastore is stopped but in node2 it was started. In node1: In node2: When checked the process id which was available in /var/run/hive/hive.pid and hive-server.pid, they are same. [node1 ~]$ cd /var/run/hive
[node1.staging.iad hive]$ ls
hive.pid hive-server.pid
[node1 hive]$ cat hive.pid
14037
[node1 hive]$ cat hive-server.pid
16654
[node1 hive]$ ps -ef| grep 14037
dinesh 564 32124 0 15:10 pts/0 00:00:00 grep 14037
hive 14037 1 0 Aug03 ? 00:01:45 /usr/lib/jvm/jre/bin/java -Xmx1024m -Dhdp.version=2.3.4.0-3485 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.4.0-3485 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.4.0-3485/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx1024m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.3.4.0-3485/hive/lib/hive-service-1.2.1.2.3.4.0-3485.jar org.apache.hadoop.hive.metastore.HiveMetaStore -hiveconf hive.log.file=hivemetastore.log -hiveconf hive.log.dir=/var/log/hive
[node1 hive]$ ps -ef| grep 16654
dinesh 621 32124 0 15:10 pts/0 00:00:00 grep 16654
hive 16654 1 0 Aug03 ? 00:01:32 /usr/lib/jvm/jre/bin/java -Xmx1024m -Dhdp.version=2.3.4.0-3485 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.4.0-3485 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.4.0-3485/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx512m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.3.4.0-3485/hive/lib/hive-service-1.2.1.2.3.4.0-3485.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
... View more
Labels:
08-02-2016
02:39 PM
When tried to connect local host with port no 2181 [node3.exp ~]$ telnet 127.0.0.1 2181
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Getting hung up. Getting same issue when using the ip address and port no of node3
... View more
07-29-2016
09:24 AM
Hi @Sebastian Carroll, Thanks for you reply. Zookeeper is running fine and listening at port 2181. [node3.exp ~]$ sudo netstat -nlp| grep 2181
tcp 0 0 :::2181 :::* LISTEN 23954/java
[node3.exp ~]$ ps 23954
PID TTY STAT TIME COMMAND
23954 ? Sl 0:42 /usr/lib/jvm/jre/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.log.file=zookeeper-zookeeper-server-node3.exp.himalaya.xyz.io.log -Dzookeeper.root.logger=INFO,ROLLIN
[node3.exp ~]$
But my Zookeeper Failover Controller is not running. Please refer it at:https://community.hortonworks.com/questions/47695/zookeeper-failover-controller-failed-to-start.html#answer-47797 Thanks
... View more
07-29-2016
09:17 AM
But there is no process running on 8031 [dev3.exp ~]$ sudo netstat -nlp| grep 8031
[dev3.exp ~]$
... View more
07-28-2016
09:37 AM
Here is the configuration file clientPort=2181
initLimit=10
autopurge.purgeInterval=24
syncLimit=5
tickTime=2000
dataDir=/disk1/hadoop/zookeeper
autopurge.snapRetainCount=30
server.1=node3.exp.dinesh.snx.io:2888:3888
server.2=node5.exp.dinesh.snx.io:2888:3888
server.3=node6.exp.dinesh.snx.io:2888:3888
... View more
07-28-2016
09:25 AM
Thanks for letting me know Artem
... View more
07-28-2016
09:25 AM
My resource manager was getting stopped automatically after some time when we started using ambari dashboard or command prompt. 2016-07-27 13:36:44,294 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=node3.exp.himalaya.xyz.io:2181,node5.exp.himalaya.xyz.io:2181,node6.exp.himalaya.xyz.io:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@77091c2f
2016-07-27 13:36:44,315 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:36:44,321 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:36:47,657 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1140)) - Client session timed out, have not heard from server in 3335ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-07-27 13:36:48,229 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.himalaya.xyz.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:36:48,230 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.himalaya.xyz.io/127.0.0.249:2181, initiating session
2016-07-27 13:36:48,230 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:36:48,379 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node6.exp.himalaya.xyz.io/127.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:36:48,379 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node6.exp.himalaya.xyz.io/127.0.0.8:2181, initiating session
2016-07-27 13:36:48,381 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1146)) - Session 0x0 for server node6.exp.himalaya.xyz.io/127.0.0.8:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)
2016-07-27 13:36:50,266 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:36:50,267 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:36:53,604 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1140)) - Client session timed out, have not heard from server in 3337ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-07-27 13:36:54,252 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.himalaya.xyz.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:36:54,252 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.himalaya.xyz.io/127.0.0.249:2181, initiating session
2016-07-27 13:36:54,252 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:36:54,311 ERROR ha.ActiveStandbyElector (ActiveStandbyElector.java:waitForZKConnectionEvent(1104)) - Connection timed out: couldn't connect to ZooKeeper in 10000 milliseconds
2016-07-27 13:36:54,355 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x0 closed
2016-07-27 13:36:54,355 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
2016-07-27 13:36:54,361 WARN ha.ActiveStandbyElector (ActiveStandbyElector.java:reEstablishSession(801)) - org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
2016-07-27 13:36:59,362 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=node3.exp.himalaya.xyz.io:2181,node5.exp.himalaya.xyz.io:2181,node6.exp.himalaya.xyz.io:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@62e5f08b
2016-07-27 13:36:59,363 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node6.exp.himalaya.xyz.io/127.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:36:59,364 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node6.exp.himalaya.xyz.io/127.0.0.8:2181, initiating session
2016-07-27 13:36:59,364 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:00,276 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:00,277 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:37:03,614 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1140)) - Client session timed out, have not heard from server in 3337ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-07-27 13:37:03,778 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.himalaya.xyz.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:03,778 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.himalaya.xyz.io/127.0.0.249:2181, initiating session
2016-07-27 13:37:03,778 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:05,439 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node6.exp.himalaya.xyz.io/127.0.0.8:2181, initiating session
2016-07-27 13:37:05,439 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:06,527 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:06,528 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:37:09,363 ERROR ha.ActiveStandbyElector (ActiveStandbyElector.java:waitForZKConnectionEvent(1104)) - Connection timed out: couldn't connect to ZooKeeper in 10000 milliseconds
2016-07-27 13:37:09,962 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x0 closed
2016-07-27 13:37:09,962 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
2016-07-27 13:37:09,962 WARN ha.ActiveStandbyElector (ActiveStandbyElector.java:reEstablishSession(801)) - org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
2016-07-27 13:37:14,963 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=node3.exp.himalaya.xyz.io:2181,node5.exp.himalaya.xyz.io:2181,node6.exp.himalaya.xyz.io:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@2dd48f21
2016-07-27 13:37:14,965 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.himalaya.xyz.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:14,965 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.himalaya.xyz.io/127.0.0.249:2181, initiating session
2016-07-27 13:37:14,965 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:15,273 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:15,274 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:37:18,611 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1140)) - Client session timed out, have not heard from server in 3337ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-07-27 13:37:19,184 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node6.exp.himalaya.xyz.io/127.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:19,185 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node6.exp.himalaya.xyz.io/127.0.0.8:2181, initiating session
2016-07-27 13:37:19,186 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:21,181 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.himalaya.xyz.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:21,181 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.himalaya.xyz.io/127.0.0.249:2181, initiating session
2016-07-27 13:37:21,181 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:22,081 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:22,082 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:37:24,965 ERROR ha.ActiveStandbyElector (ActiveStandbyElector.java:waitForZKConnectionEvent(1104)) - Connection timed out: couldn't connect to ZooKeeper in 10000 milliseconds
2016-07-27 13:37:25,516 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x0 closed
2016-07-27 13:37:25,516 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
2016-07-27 13:37:25,516 WARN ha.ActiveStandbyElector (ActiveStandbyElector.java:reEstablishSession(801)) - org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
2016-07-27 13:37:30,517 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=node3.exp.himalaya.xyz.io:2181,node5.exp.himalaya.xyz.io:2181,node6.exp.himalaya.xyz.io:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@339934ae
2016-07-27 13:37:30,518 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:30,519 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:37:33,856 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1140)) - Client session timed out, have not heard from server in 3337ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-07-27 13:37:34,690 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.himalaya.xyz.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:34,690 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.himalaya.xyz.io/127.0.0.249:2181, initiating session
2016-07-27 13:37:34,691 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:35,046 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node6.exp.himalaya.xyz.io/127.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:35,047 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node6.exp.himalaya.xyz.io/127.0.0.8:2181, initiating session
2016-07-27 13:37:35,048 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:36,508 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:36,509 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:37:39,844 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1140)) - Client session timed out, have not heard from server in 3335ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-07-27 13:37:40,276 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.himalaya.xyz.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:40,277 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.himalaya.xyz.io/127.0.0.249:2181, initiating session
2016-07-27 13:37:40,277 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:40,518 ERROR ha.ActiveStandbyElector (ActiveStandbyElector.java:waitForZKConnectionEvent(1104)) - Connection timed out: couldn't connect to ZooKeeper in 10000 milliseconds
2016-07-27 13:37:40,763 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x0 closed
2016-07-27 13:37:40,763 WARN ha.ActiveStandbyElector (ActiveStandbyElector.java:reEstablishSession(801)) - org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
2016-07-27 13:37:45,764 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=node3.exp.himalaya.xyz.io:2181,node5.exp.himalaya.xyz.io:2181,node6.exp.himalaya.xyz.io:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@fa07419
2016-07-27 13:37:45,766 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:45,767 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:37:49,102 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1140)) - Client session timed out, have not heard from server in 3335ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-07-27 13:37:49,929 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node6.exp.himalaya.xyz.io/127.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:49,930 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node6.exp.himalaya.xyz.io/127.0.0.8:2181, initiating session
2016-07-27 13:37:49,930 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:50,680 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.himalaya.xyz.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:50,680 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.himalaya.xyz.io/127.0.0.249:2181, initiating session
2016-07-27 13:37:50,681 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 13:37:52,450 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:37:52,451 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:37:55,765 ERROR ha.ActiveStandbyElector (ActiveStandbyElector.java:waitForZKConnectionEvent(1104)) - Connection timed out: couldn't connect to ZooKeeper in 10000 milliseconds
2016-07-27 13:37:55,885 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x0 closed
2016-07-27 13:37:55,885 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
2016-07-27 13:37:55,886 WARN ha.ActiveStandbyElector (ActiveStandbyElector.java:reEstablishSession(801)) - org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
2016-07-27 13:38:00,887 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=node3.exp.himalaya.xyz.io:2181,node5.exp.himalaya.xyz.io:2181,node6.exp.himalaya.xyz.io:2181 sessionTimeout=10000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@6e0e2eec
2016-07-27 13:38:00,888 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.himalaya.xyz.io/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 13:38:00,889 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.himalaya.xyz.io/127.0.0.1:2181, initiating session
2016-07-27 13:38:04,226 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1140)) - Client session timed out, have not heard from server in 3337ms for sessionid 0x0, closing socket connection and attempting reconnect
Thanks.
... View more
Labels:
07-27-2016
03:19 PM
zkFailoverController is getting stopped automatically after restarting it from ambari/command prompt. Below is the log: All the required configuration like dfs.ha.automatic-failover.enabled and ha.zookeeper.quorum are set 2016-07-27 14:15:08,334 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.library.path=:/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native:/usr/hdp/2.3.4.0-3485/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native
2016-07-27 14:15:08,335 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.tmpdir=/tmp
2016-07-27 14:15:08,335 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.compiler=<NA>
2016-07-27 14:15:08,335 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.name=Linux
2016-07-27 14:15:08,335 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.arch=amd64
2016-07-27 14:15:08,335 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.version=2.6.32-573.8.1.el6.x86_64
2016-07-27 14:15:08,335 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.name=hdfs
2016-07-27 14:15:08,337 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.home=/home/hdfs
2016-07-27 14:15:08,337 INFO zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.dir=/usr/hdp/2.3.4.0-3485/hadoop
2016-07-27 14:15:08,338 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=node3.exp.dinesh.snx.io:2181,node5.exp.dinesh.snx.io:2181,node6.exp.dinesh.snx.io:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@2fbb21e
2016-07-27 14:15:08,359 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.dinesh.snx.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 14:15:08,369 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.dinesh.snx.io/127.0.0.249:2181, initiating session
2016-07-27 14:15:08,370 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 14:15:09,417 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.dinesh.snx.io/127.0.0.7:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 14:15:09,418 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.dinesh.snx.io/127.0.0.7:2181, initiating session
2016-07-27 14:15:11,087 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1140)) - Client session timed out, have not heard from server in 1669ms for sessionid 0x0, closing socket connection and attempting reconnect
2016-07-27 14:15:11,442 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node6.exp.dinesh.snx.io/127.0.0.8:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 14:15:11,443 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node6.exp.dinesh.snx.io/127.0.0.8:2181, initiating session
2016-07-27 14:15:11,443 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 14:15:12,643 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node3.exp.dinesh.snx.io/127.0.0.249:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 14:15:12,644 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node3.exp.dinesh.snx.io/127.0.0.249:2181, initiating session
2016-07-27 14:15:12,644 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-07-27 14:15:13,355 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server node5.exp.dinesh.snx.io/127.0.0.7:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 14:15:13,356 ERROR ha.ActiveStandbyElector (ActiveStandbyElector.java:waitForZKConnectionEvent(1104)) - Connection timed out: couldn't connect to ZooKeeper in 5000 milliseconds
2016-07-27 14:15:13,356 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established to node5.exp.dinesh.snx.io/127.0.0.7:2181, initiating session
2016-07-27 14:15:15,126 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x0 closed
2016-07-27 14:15:15,126 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
2016-07-27 14:15:15,132 FATAL ha.ZKFailoverController (ZKFailoverController.java:doRun(193)) - Unable to start failover controller. Unable to connect to ZooKeeper quorum at zk3.example.com:2181,zk5.example.com:2181,zk6.example.com:2181. Please check the configured value for ha.zookeeper.quorum and ensure that ZooKeeper is running.
2016-07-27 14:15:15,135 INFO tools.DFSZKFailoverController (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at node4.exp.dinesh.snx.io/127.0.0.6
***********************************************************
Thanks
... View more
Labels:
07-27-2016
12:40 PM
After starting the nodemanager, it was getting stopped when tried to connect with one of the resource managers by throwing the below. 2016-07-27 11:03:03,820 INFO retry.RetryInvocationHandler (RetryInvocationHandler.java:invoke(144)) - Exception while invoking registerNodeManager of class ResourceTrackerPBClientImpl over rm2 after 17 fail over attempts. Trying to fail over after sleeping for 34825ms.
java.net.ConnectException: Call From dev5.exp.caspian.rax.io to dev3.exp.caspian.rax.io:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
... View more
07-07-2016
11:43 AM
Thanks for you reply @Dileep Kumar Chiguruvada. But my doubt here is that can we use HDFS location for setting the environment variable, bcse in general we will use only linux file system path to set environment variables. If we use HDFS location, then how the OS will understand that it is a linux file system or HDFS file system?
... View more
07-05-2016
09:48 AM
I want all the jars files like csv jar and json jar should be in hdfs location by that it will load the jar files easily when hive cli is loaded? My question here is, Can we use HDFS path for setting environment variable like below: export HIVE_AUX_JARS_PATH=hdfs:///hdp/apps/2.3.4.0-3485/hive/auxjars/*
Does it will load the jar files when we used the HIVE_AUX_JARS_PATH?
... View more
Labels:
