Hue kerberos authentication - bad format in credentials cache

[#1]

Hello,

I have a KDC sitting on an existing Active Directory instance and have successfully installed kerberos in Hadoop and have been using this successfully with the command line.

However, I cannot get it working with Hue.

I have HA configured therefore have installed and configured Hadoop-HTTPFS. This is working via command line:

hue@edge:~$ kinit -kt /etc/security/keytabs/hue.service.keytab hue
hue@edge:~$ curl -i --negotiate -u : "http://edge:14000/webhdfs/v1/user/?op=LISTSTATUS"
HTTP/1.1 200 OK
......{"FileStatuses":{"FileStatus":[{"pathSuffix":"ambari-qa",...... etc 

However when I log into Hue, I first see the "Cannot create home directory" error and then when I try to access the Filebrowser tab, this appears in the server logs:

[04/Jul/2016 08:45:32 -0700] middleware   INFO     Processing exception: Cannot access: /user/daleb.  Note: you are a Hue admin but not a HDFS superuser, "hdfs" or part of HDFS supergroup, "hdfs".: Traceback (most recent call last):
  File "/opt/hue/build/env/local/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/hue/build/env/local/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/transaction.py", line 371, in inner
    return func(*args, **kwargs)
  File "/opt/hue/apps/filebrowser/src/filebrowser/views.py", line 108, in index
    return view(request, path)
  File "/opt/hue/apps/filebrowser/src/filebrowser/views.py", line 186, in view
    raise PopupException(msg , detail=e)
PopupException: Cannot access: /user/daleb.  Note: you are a Hue admin but not a HDFS superuser, "hdfs" or part of HDFS supergroup, "hdfs".
[04/Jul/2016 08:45:32 -0700] webhdfs      ERROR    Failed to determine superuser of WebHdfs at http://edge:14000/webhdfs/v1/: Unable to authenticate <Response [401]>
Traceback (most recent call last):
  File "/opt/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 149, in superuser
    sb = self.stats('/')
  File "/opt/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 236, in stats
    res = self._stats(path)
  File "/opt/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 230, in _stats
    raise ex
WebHdfsException: Unable to authenticate <Response [401]>
[04/Jul/2016 08:45:32 -0700] kerberos_    ERROR    handle_mutual_auth(): Mutual authentication failed
[04/Jul/2016 08:45:32 -0700] kerberos_    ERROR    authenticate_server(): authGSSClientStep() failed:
Traceback (most recent call last):
  File "/opt/hue/build/env/local/lib/python2.7/site-packages/requests_kerberos-0.6.1-py2.7.egg/requests_kerberos/kerberos_.py", line 229, in authenticate_server
    _negotiate_value(response))
GSSError: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Bad format in credentials cache', -1765328185))

I have the proxy user configurations set up.

I have created the `hue_krb5_ccache` file but also notice this:

hue@edge:/tmp$ klist -k hue_krb5_ccache
Keytab name: FILE:hue_krb5_ccache
klist: Unsupported key table format version number while starting keytab scan

hue@edge:/tmp# kinit -f -c hue_krb5_ccache
klist: Bad format in credentials cache while setting cache flags (ticket cache FILE:/tmp/hue_krb5_ccache)

Does anyone have any suggestions? I've seen few things regarding the kt_renewer but not quite sure where that fits in with my architecture.

Thanks.

[#2]

@Dale Bradman

I believe you have working kerberos configuration, by 'hue' user, you are getting list of files from '/user/' on hdfs which is expected as other users can list the files under /user/ from hdfs.

When you login to hue UI, it is trying to create directory under /user/ on hdfs which only 'hdfs' user can do. Can you please create /user/<user-name> directory on HDFS by 'hdfs' user and then try to login via HUE UI again?

[#3]

Thanks @Kuldeep Kulkarni, but I have already created this folder and given it 777 permissions just to be certainly sure. The same error messages still exist.

On the About Hue page there is the failed to access filesystem root error appearing.

The Hue.ini file is 100% correct, I have validated this many times.

Could you please elaborate on how Hue accesses the cluster using kerberos?

1. Proxy user configs are set to allow the Hue user to access the cluster on other user's behalf. 2. User logs into Hue however only Hue is required to be authenticated using kerberos because the proxy configs are set. 3. Once Hue is authenticated, it accesses the cluster as a proxy user but authorization is still applied on the user level.

Is HDFS ever required to be authenticated during this process?

Also, is there anything I need to watch out for with Ranger?

Thanks.

[#4]

@Dale Bradman, I believe you are using an incorrect option for klist to list your credential cache:

hue@edge:/tmp$ klist -k hue_krb5_ccache
Keytab name: FILE:hue_krb5_ccache
klist: Unsupported key table format version number while starting keytab scan

The -k option means "List keys held in a keytab file." You are trying to list the contents of a cache file, not a keytab file. Without the "-k" option you should get some more usable information

[root@c6501 ~]# klist  /tmp/krb5cc_1002
Ticket cache: FILE:/tmp/krb5cc_1002
Default principal: hdfs-c1@EXAMPLE.COM


Valid starting     Expires            Service principal
07/05/16 14:18:58  07/06/16 14:18:58  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 07/05/16 14:18:58

[#5]

Yes my mistake. The thing is I'm not entirely sure on the purpose of the /tmp/hue_krb5_ccache file - should there be a setting in the hue.ini file that defines this? I created the file because Hue complained it did not exist in the "About Hue" page.

hue@edge:~$ klist /tmp/hue_krb5_ccache
klist: Bad format in credentials cache while setting cache flags (ticket cache FILE:/tmp/hue_krb5_ccache)

Do you know how much of an impact Ranger has on this? For example, I cannot see the Hue user in the Ranger UI.

Thanks.

[#6]

I am not too familiar with Hue or Ranger.

Have you followed the steps in https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_installing_manually_book/content/ref-143... in order to allow Hue to authenticate using Kerberos?

[#7]

Yes. Thanks for the help anyways

[#8]

@Robert Levas

Hello,

I'm facing a similar issue with Hue. I've detailedly explained my question here:

https://community.hortonworks.com/questions/147826/failed-to-access-filesystem-root-through-hue-ui.h....

The output of $klist with hue user is mentioned in klist.png.

Could you please help me to resolve this issue? I'll be really grateful.

[#9]

Hi,Dale Bradman，I have the same mistakes as you, do you ask? Can you help me? Thank you!

[#10]

This question bothers me for 2 days! ：）