Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBASE ZOOKEEPER CONFIG CDH5.4.4

SOLVED Go to solution
Highlighted

HBASE ZOOKEEPER CONFIG CDH5.4.4

New Contributor

I have a Cluster with 2 NameNodes and 8 DataNodes

 

Zookeeper is Configured on datanodes 04,05 and 06 . 

 

All my datanodes are Hbase Region servers and the namenodes are Hmasters .

 

Use case : I'm connecting to Hbase through a Java API which does an UPSERT logic in Reducer . 

 

If the reduce tasks kicks off any other node apart from ZOOKEEPER nodes my tasks fail with exception .

 

got the same zookeeper issue

2015-08-12 19:46:35,475 WARN [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
2015-08-12 19:46:35,475 INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-12 19:46:35,475 INFO [main] org.apache.hadoop.hbase.util.RetryCounter: Sleeping 8000ms before retry #3...
2015-08-12 19:46:35,476 WARN [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused

 

 

 

I have my hbase-site.xml configured on all nodes as 

 

<property>
<name>zookeeper.znode.parent</name>
<value>/hbase</value>
</property>
<property>
<name>zookeeper.znode.rootserver</name>
<value>root-region-server</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>pc1ushpdhad06.res.prod.global,pc1ushpdhad04.res.prod.global,pc1ushpdhad05.res.prod.global</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>

 

 

Can you please assist and also let me know if I have to move hbase-site.xml to /opt/cloudera/parcels/CDH/lib/hadoop-yarn or anyother directory .

 

I tried passing the conf at Java API , without any luck

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HBASE ZOOKEEPER CONFIG CDH5.4.4

Master Guru
How exactly are you invoking your Java application? Is /etc/hbase/conf/ directory on the classpath?

If the failure you're quoting is from within the Reducer task log, then ensure you don't do a
"new Configuration()"/"HBaseConfiguration.create()" anywhere inside the Reducer class, but instead reuse context.getConfiguration() to obtain the standard job configuration which will carry the ZK properties.
2 REPLIES 2

Re: HBASE ZOOKEEPER CONFIG CDH5.4.4

Master Guru
How exactly are you invoking your Java application? Is /etc/hbase/conf/ directory on the classpath?

If the failure you're quoting is from within the Reducer task log, then ensure you don't do a
"new Configuration()"/"HBaseConfiguration.create()" anywhere inside the Reducer class, but instead reuse context.getConfiguration() to obtain the standard job configuration which will carry the ZK properties.

Re: HBASE ZOOKEEPER CONFIG CDH5.4.4

New Contributor

Thank you Harsh . This solution worked for me .

 

I have 21000 records at my source and I'm doing a UPSERT in Hbase using Java API . I'm not sure of Hbase is making additional connections to Zookeeper and each such instance is a FileDescriptor . Attached part of log . It took 10 minutes for the Job to complete for a Job with 4 reduces tasks .

 

2015-08-17 16:07:12,577 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2279f558 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,577 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x2279f5580x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,579 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49332, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,580 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863f8, negotiated timeout = 60000
2015-08-17 16:07:12,584 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863f8
2015-08-17 16:07:12,585 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863f8 closed
2015-08-17 16:07:12,585 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,685 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5cdeb23b connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,685 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x5cdeb23b0x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,688 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49335, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,689 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fa, negotiated timeout = 60000
2015-08-17 16:07:12,693 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fa
2015-08-17 16:07:12,694 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fa closed
2015-08-17 16:07:12,694 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,795 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1dbc8040 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,795 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x1dbc80400x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,798 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,799 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49339, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,800 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fc, negotiated timeout = 60000
2015-08-17 16:07:12,803 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fc
2015-08-17 16:07:12,804 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fc closed
2015-08-17 16:07:12,804 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-08-17 16:07:12,905 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x64f98ef1 connecting to ZooKeeper ensemble=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181
2015-08-17 16:07:12,905 INFO [main] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181 sessionTimeout=60000 watcher=hconnection-0x64f98ef10x0, quorum=pc1ushpdhad04.res.prod.global:2181,pc1ushpdhad06.res.prod.global:2181,pc1ushpdhad05.res.prod.global:2181, baseZNode=/hbase
2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Opening socket connection to server pc1ushpdhad06.res.prod.global/10.210.34.118:2181. Will not attempt to authenticate using SASL (unknown error)
2015-08-17 16:07:12,908 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.210.34.120:49342, server: pc1ushpdhad06.res.prod.global/10.210.34.118:2181
2015-08-17 16:07:12,909 INFO [main-SendThread(pc1ushpdhad06.res.prod.global:2181)] org.apache.zookeeper.ClientCnxn: Session establishment complete on server pc1ushpdhad06.res.prod.global/10.210.34.118:2181, sessionid = 0x14f29129af863fe, negotiated timeout = 60000
2015-08-17 16:07:12,912 INFO [main] org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14f29129af863fe
2015-08-17 16:07:12,914 INFO [main] org.apache.zookeeper.ZooKeeper: Session: 0x14f29129af863fe closed