Member since
08-13-2015
2
Posts
0
Kudos Received
0
Solutions
08-17-2015
09:38 AM
Thank you Harsh . This solution worked for me . I have 21000 records at my source and I'm doing a UPSERT in Hbase using Java API . I'm not sure of Hbase is making additional connections to Zookeeper and each such instance is a FileDescriptor . Attached part of log . It took 10 minutes for the Job to complete for a Job with 4 reduces tasks . 2015-08-17 16:07:12,577 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2279f558 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,577 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x2279f5580x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49332, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,580 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863f8, negotiated timeout = 60000
2015-08-17 16:07:12,584 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863f8
2015-08-17 16:07:12,585 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863f8 closed
2015-08-17 16:07:12,585 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,685 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5cdeb23b connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,685 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x5cdeb23b0x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49335, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,689 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fa, negotiated timeout = 60000
2015-08-17 16:07:12,693 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fa
2015-08-17 16:07:12,694 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fa closed
2015-08-17 16:07:12,694 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,795 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1dbc8040 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,795 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x1dbc80400x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,798 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,799 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49339, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,800 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fc, negotiated timeout = 60000
2015-08-17 16:07:12,803 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fc
2015-08-17 16:07:12,804 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fc closed
2015-08-17 16:07:12,804 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,905 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x64f98ef1 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,905 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x64f98ef10x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49342, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,909 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fe, negotiated timeout = 60000
2015-08-17 16:07:12,912 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fe
2015-08-17 16:07:12,914 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fe closed
... View more
08-13-2015
08:38 PM
I have a Cluster with 2 NameNodes and 8 DataNodes Zookeeper is Configured on datanodes 04,05 and 06 . All my datanodes are Hbase Region servers and the namenodes are Hmasters . Use case : I'm connecting to Hbase through a Java API which does an UPSERT logic in Reducer . If the reduce tasks kicks off any other node apart from ZOOKEEPER nodes my tasks fail with exception . got the same zookeeper issue 2015-08-12 19:46:35,475 WARN [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server 2015-08-12 19:46:35,475 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2015-08-12 19:46:35,475 INFO [main] org.apache.hadoop.hbase.util.RetryCounter: Sleeping 8000ms before retry #3... 2015-08-12 19:46:35,476 WARN [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused I have my hbase-site.xml configured on all nodes as <property> <name>zookeeper.znode.parent</name> <value>/hbase</value> </property> <property> <name>zookeeper.znode.rootserver</name> <value>root-region-server</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>pc1ushpdhad06.res.prod.global,pc1ushpdhad04.res.prod.global,pc1ushpdhad05.res.prod.global</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> </property> Can you please assist and also let me know if I have to move hbase-site.xml to /opt/cloudera/parcels/CDH/lib/hadoop-yarn or anyother directory . I tried passing the conf at Java API , without any luck
... View more