Skip to main content
Murthy
Murthy
Gold ⭐️
April 21, 2014
Solved

Fix HBase NoServerForRegionException in Kerberized Oozie Workflows

Hello,

We are trying to execute a MapReduce job to write to HBASE using OOZIE Workflow.

We have implemented Kerberos in our Cluster and that is when these issues started....

The Job is failing after 10 minutes with an error:

Screenshot 2026-05-14 at 9.39.14 AM.png

We had a similar issue with beeline not able to run HBASE Commands and the solution was -

I think the issue is that HBase does not allow hive to impersonate users.  So you'll need to setup hive as a proxy user in HBase.  Can you try the following:

- Go to the HDFS service configuration in CM.

- Go to Service-Wide->Advanced and add the following to "Cluster-wide Configuration Safety Valve for core-site.xml":

Screenshot 2026-05-14 at 9.05.20 AM.png

I have added similar entry for OOZIE - HDFS Service-Wide Advanced properties and add the following to Cluster-wide ConfigurationSafety Valve for core-site.xml

Screenshot 2026-05-14 at 9.08.13 AM.png

Here is what i have in the logs -

Screenshot 2026-05-14 at 9.28.25 AM.png

Oozie Launcher failed, finishing Hadoop job gracefully

Murthy
MurthyAnswer
Gold ⭐️
May 5, 2014

Hello Clint, I followed the directions in an article told by Ryan Geno (Cloudera Employee) and it seems to have fixed the issue - 

http://prodlife.wordpress.com/2013/11/22/using-oozie-in-kerberized-cluster/

 

- go to “Oozie service->Configuration->Oozie Server(default)->Advanced-> Oozie Server Configuration Safety Valve for oozie-site.xml” and add:

1

2

3

4

<property>

<name>oozie.credentials.credentialclasses</name>

<value>hcat=org.apache.oozie.action.hadoop.HCatCredentials</value>

</property>

 

 

After I restarted the Oozie Service and ran the ResourceClipper Job – it completed….

 

Thanks Murthy

View in thread

5 replies

Murthy
MurthyAuthor
Gold ⭐️
April 21, 2014

Here is the information from the syslog - 

 

the syslog shows hundreds of the following errors until the job dies:

 

2014-04-21 08:44:26,676 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

2014-04-21 08:44:26,676 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)

at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)

2014-04-21 08:44:26,686 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

2014-04-21 08:44:26,686 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 2000ms before retry #1...

Translate
Clint
Platinum ⭐️
Clint
Platinum ⭐️
April 23, 2014

Murthy,

 

  If this is a CM-managed cluster, you need to deploy your HBase Client Configuration to any node that will be submitting jobs that will communicate with HBase.  It looks like the machine where the job is running is trying to connect to a zookeeeper instance on it's local machine (the default) and it can't find zookeeper and therefore cannot communicate with HBase.  The hbase-site.xml file loaded in this client's environment path will solve that.

Translate
Murthy
MurthyAuthor
Gold ⭐️
April 25, 2014

Hello Clint, Yes we are using CDH 4.6...

 

I have copied the hbase-site.xml to /user/oozie/share/lib/hbase folder in HDFS.

 

I got the error - i have restarted the cluster also... Here are some lines from the syslog - 

 

2014-04-25 13:19:41,310 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
2014-04-25 13:19:41,331 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 42965@ldxhdfsw4.dx.deere.com
2014-04-25 13:19:41,337 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2014-04-25 13:19:41,339 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused

 

Murthy

Translate
Terms & ConditionsAccessibility statement