Member since
08-13-2015
2
Posts
0
Kudos Received
0
Solutions
08-17-2015
09:38 AM
Thank you Harsh . This solution worked for me . I have 21000 records at my source and I'm doing a UPSERT in Hbase using Java API . I'm not sure of Hbase is making additional connections to Zookeeper and each such instance is a FileDescriptor . Attached part of log . It took 10 minutes for the Job to complete for a Job with 4 reduces tasks . 2015-08-17 16:07:12,577 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2279f558 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,577 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x2279f5580x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49332, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,580 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863f8, negotiated timeout = 60000
2015-08-17 16:07:12,584 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863f8
2015-08-17 16:07:12,585 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863f8 closed
2015-08-17 16:07:12,585 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,685 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5cdeb23b connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,685 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x5cdeb23b0x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49335, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,689 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fa, negotiated timeout = 60000
2015-08-17 16:07:12,693 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fa
2015-08-17 16:07:12,694 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fa closed
2015-08-17 16:07:12,694 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,795 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1dbc8040 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,795 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x1dbc80400x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,798 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,799 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49339, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,800 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fc, negotiated timeout = 60000
2015-08-17 16:07:12,803 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fc
2015-08-17 16:07:12,804 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fc closed
2015-08-17 16:07:12,804 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,905 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x64f98ef1 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,905 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x64f98ef10x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49342, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,909 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fe, negotiated timeout = 60000
2015-08-17 16:07:12,912 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fe
2015-08-17 16:07:12,914 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fe closed
... View more