HBASE ZOOKEEPER CONFIG CDH5.4.4

[#1]

I have a Cluster with 2 NameNodes and 8 DataNodes

Zookeeper is Configured on datanodes 04,05 and 06 . 

All my datanodes are Hbase Region servers and the namenodes are Hmasters .

Use case : I'm connecting to Hbase through a Java API which does an UPSERT logic in Reducer . 

If the reduce tasks kicks off any other node apart from ZOOKEEPER nodes my tasks fail with exception .

got the same zookeeper issue

2015-08-12 19:46:35,475 WARN [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
2015-08-12 19:46:35,475 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-12 19:46:35,475 INFO [main] org.apache.hadoop.hbase.util.RetryCounter: Sleeping 8000ms before retry #3...
2015-08-12 19:46:35,476 WARN [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused

I have my hbase-site.xml configured on all nodes as 

<property>
<name>zookeeper.znode.parent</name>
<value>/hbase</value>
</property>
<property>
<name>zookeeper.znode.rootserver</name>
<value>root-region-server</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>pc1ushpdhad06.res.prod.global,pc1ushpdhad04.res.prod.global,pc1ushpdhad05.res.prod.global</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>

Can you please assist and also let me know if I have to move hbase-site.xml to /opt/cloudera/parcels/CDH/lib/hadoop-yarn or anyother directory .

I tried passing the conf at Java API , without any luck

[#2 · Accepted solution]

How exactly are you invoking your Java application? Is /etc/hbase/conf/ directory on the classpath?

If the failure you're quoting is from within the Reducer task log, then ensure you don't do a
"new Configuration()"/"HBaseConfiguration.create()" anywhere inside the Reducer class, but instead reuse context.getConfiguration() to obtain the standard job configuration which will carry the ZK properties.

[#3]

Thank you Harsh . This solution worked for me .

I have 21000 records at my source and I'm doing a UPSERT in Hbase using Java API . I'm not sure of Hbase is making additional connections to Zookeeper and each such instance is a FileDescriptor . Attached part of log . It took 10 minutes for the Job to complete for a Job with 4 reduces tasks .

2015-08-17 16:07:12,577 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2279f558 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,577 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x2279f5580x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49332, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,580 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863f8, negotiated timeout = 60000
2015-08-17 16:07:12,584 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863f8
2015-08-17 16:07:12,585 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863f8 closed
2015-08-17 16:07:12,585 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,685 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5cdeb23b connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,685 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x5cdeb23b0x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49335, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,689 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fa, negotiated timeout = 60000
2015-08-17 16:07:12,693 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fa
2015-08-17 16:07:12,694 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fa closed
2015-08-17 16:07:12,694 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,795 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1dbc8040 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,795 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x1dbc80400x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,798 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,799 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49339, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,800 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fc, negotiated timeout = 60000
2015-08-17 16:07:12,803 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fc
2015-08-17 16:07:12,804 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fc closed
2015-08-17 16:07:12,804 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,905 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x64f98ef1 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,905 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x64f98ef10x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49342, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,909 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fe, negotiated timeout = 60000
2015-08-17 16:07:12,912 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fe
2015-08-17 16:07:12,914 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fe closed