Support Questions
Find answers, ask questions, and share your expertise

"Api Error:" in Hue When Using HBase Browser for Secure HBase Cluster

"Api Error:" in Hue When Using HBase Browser for Secure HBase Cluster

Good day, hope this message finds you well. We enabled HBase a few weeks back, but just took the defaults which don't enable Authentication or Authorization. We just wanted to do a quick POC.

 

We are now going back and implementing Authentication and Authorization as one of the POC projects has PHI data and so we want to make sure we protect that.

 

We are using CDH/CM 5.4 and have went through the follow two articles and seem to have everything working for our application. However, we are running into a problem with Hue's HBase Browser.

 

Authentication = http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_sg_hbase_authen...
Authorization = http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_sg_hbase_author...

 

In addition to making the HBase changes, we have also went through this article and made these changes for Hue.

 

Changes for Hue = http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/admin_hue_enable_ap...

 

In the HBase Browser we get the error "Api Error:" that pops up in a little red box. Below are the errors we get in Hue and the ones we get on our Thrift server. It seems to boil down to the message 'Authorization header received from the client is empty.'.

 

It kind of looks like someone had a similar issue in the comments of this article, but I can't seem to tell what they exactly did to fix it.

 

http://gethue.com/hbase-browsing-with-doas-impersonation-and-kerberos/

 

Any other things you can think of that we should be looking at?

 

Thanks in advance,

 

Mac

 

#here is the error message we are getting in the Hue logs when trying to access the HBase Browswer after enabling Authentication and Authorization in HBase.
Nov 4, 1:59:31 AM INFO access
10.42.63.12 NolandM - "POST /hbase/api/getTableList/HBase HTTP/1.1"
Nov 4, 1:59:31 AM INFO connectionpool
Resetting dropped connection: mapls189.bsci.bossci.com
Nov 4, 1:59:31 AM ERROR kerberos_
handle_mutual_auth(): Mutual authentication unavailable on 413 response
Nov 4, 1:59:31 AM ERROR thrift_util
Thrift saw exception (this may be expected).
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/desktop/core/src/desktop/lib/thrift_util.py", line 415, in wrapper
ret = res(*args, **kwargs)
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/apps/hbase/src/hbase/../../gen-py/hbased/Hbase.py", line 53, in decorate
return func(*args, **kwargs)
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/apps/hbase/src/hbase/../../gen-py/hbased/Hbase.py", line 832, in getTableNames
self.send_getTableNames()
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/apps/hbase/src/hbase/../../gen-py/hbased/Hbase.py", line 840, in send_getTableNames
self._oprot.trans.flush()
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/build/env/lib/python2.6/site-packages/thrift-0.9.1-py2.6-linux-x86_64.egg/thrift/transport/TTransport.py", line 170, in flush
self.__trans.flush()
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/desktop/core/src/desktop/lib/thrift_/http_client.py", line 84, in flush
self._data = self._root.post('', data=data, headers=self._headers)
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 122, in post
return self.invoke("POST", relpath, params, data, self._make_headers(contenttype, headers))
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 78, in invoke
urlencode=self._urlencode)
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/desktop/core/src/desktop/lib/rest/http_client.py", line 161, in execute
raise self._exc_class(ex)
RestException: (error 413)
Nov 4, 1:59:31 AM INFO thrift_util
Thrift saw exception: (error 413)
Nov 4, 1:59:31 AM INFO middleware
Processing exception: Api Error: : Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/handlers/base.py", line 112, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/transaction.py", line 371, in inner
return func(*args, **kwargs)
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/apps/hbase/src/hbase/views.py", line 76, in api_router
return api_dump(HbaseApi(request.user).query(*url_params))
File "/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/hue/apps/hbase/src/hbase/api.py", line 54, in query
raise PopupException(_("Api Error: %s") % e.message)
PopupException: Api Error:


####### Here is the error we see on the Thrift server side.

Nov 4, 1:59:31.238 PM WARN org.apache.hadoop.security.UserGroupInformation
PriviledgedActionException as:HTTP/ourhost@ourdomain.COM (auth:KERBEROS) cause:org.apache.hadoop.hbase.thrift.HttpAuthenticationException: Authorization header received from the client is empty.

Nov 4, 1:59:31.239 PM ERROR org.apache.hadoop.hbase.thrift.ThriftHttpServlet
Failed to perform authentication

Nov 4, 1:59:31.239 PM ERROR org.apache.hadoop.hbase.thrift.ThriftHttpServlet
Kerberos Authentication failed
org.apache.hadoop.hbase.thrift.HttpAuthenticationException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.hbase.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:139)
at org.apache.hadoop.hbase.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:86)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1684)
at org.apache.hadoop.hbase.thrift.ThriftHttpServlet.doKerberosAuth(ThriftHttpServlet.java:134)
... 16 more
Caused by: org.apache.hadoop.hbase.thrift.HttpAuthenticationException: Authorization header received from the client is empty.
at org.apache.hadoop.hbase.thrift.ThriftHttpServlet$HttpKerberosServerAction.getAuthHeader(ThriftHttpServlet.java:212)
at org.apache.hadoop.hbase.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:176)
at org.apache.hadoop.hbase.thrift.ThriftHttpServlet$HttpKerberosServerAction.run(ThriftHttpServlet.java:144)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
... 17 more

 

2 REPLIES 2

Re: "Api Error:" in Hue When Using HBase Browser for Secure HBase Cluster

Some potential hints

If you are getting this error:
1
Caused by: org.apache.hadoop.hbase.thrift.HttpAuthenticationException:
Authorization header received from the client is empty.

You are very probably hitting
https://issues.apache.org/jira/browse/HBASE-13069. Also make sure the
HTTP/_HOST principal is in the keytab of for their HBase Thrift Server.
Beware that as a follow-up you might get
https://issues.apache.org/jira/browse/HBASE-14471.


Re: "Api Error:" in Hue When Using HBase Browser for Secure HBase Cluster

Thanks for the reply.   We're running CDH 5.4.5 which is on HBase 1.0.0 so indeed maybe we are running into those two issues - unless they were back ported into CDH.

 

After enabling debug on the Thrift server, we see this message in the logs so seems like we're onto something with the second issue for sure.

 

We are also having issues with the REST interface, which seems to be related.

 

I think we're going to submit a ticket to see if we can get some help from support.  I appreciate your help in pointing us to the right direction.

 

All the best,

 

Mac

 

 

#from curl command to REST interface.

 

-bash-4.1$ curl -X GET -i --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://mapls189:20550/version/cluster
HTTP/1.1 401 Authentication required
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
Content-Type: text/html; charset=iso-8859-1
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1408

HTTP/1.1 413 FULL head
Connection: close

 

#from Thrift logs

 

EXCEPTION 
HttpException(413,FULL head,null)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:285)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)