Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Job browser in hue does not work for Kerberos and Map Reduce

Job browser in hue does not work for Kerberos and Map Reduce

New Contributor

On HDP 2.4, we're using the standard Hue with security on (Hue 2.6.1-258)

We have map reduce, pig, hive and spark running successfully from the command line and Kerberos.

In Hue, we have also everything working fine, expect when we submit a Map Reduce job: the job is submitted and working, but when looking at it from the job history, we are getting the following error:

Job job_1474295224072_0004 could not be found: <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> <title>Error 401 Authentication required</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /ws/v1/history/mapreduce/jobs/job_1474295224072_0004. Reason: <pre> Authentication required</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> </body> </html> (error 401)

When submitting TEZ or Spark jobs, there is no issue, we can see the job details ( it's actually going to another REST url to query for information). The job browser problem is only limited to Map Reduce jobs.

When tracking down the issue, we see the following URL in the Hue logs for the job history:

http_client  DEBUG    REST invocation: curl -X GET  -H Accept: application/json 'http://my_cluster:19888/ws/v1/history/mapreduce/jobs/job_1474295224072_0004' 

Which is wrong. It should contain a --negotiate such as:

http_client  DEBUG    REST invocation: curl -X GET --negotiate -H Accept: application/json 'http://my_cluster:19888/ws/v1/history/mapreduce/jobs/job_1474295224072_0004' 

Looking at the source code shows no indication of any kind of support for that.

So my question is: is Hue shipped in HDP 2.4 capable to use Kerberos for this job browser? If not, any hints on what is wrong in our setup?

7 REPLIES 7

Re: Job browser in hue does not work for Kerberos and Map Reduce

Contributor

There could be a possible misconfiguration in your hue.ini. Please open /etc/hue/conf/hue.ini and ensure the value of history_server_api_url, i.e. the host name (most of the times is same as the resource manager host) and the port number is correct. Also looking up logs at /var/log/hue/runcpserver.log should be helpful.

Re: Job browser in hue does not work for Kerberos and Map Reduce

New Contributor

Thanks for the suggestion.

I am confirming that history_server_api_url is correct.

Looking at the logs you are referring reveals the following when looking at a Map Reduce job:

[20/Sep/2016 08:45:15 +0000] access       INFO     10.50.0.59 trifil - "GET /jobbrowser/jobs/application_1474295224072_0013 HTTP/1.1"
[20/Sep/2016 08:45:15 +0000] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[20/Sep/2016 08:45:15 +0000] middleware   INFO     Processing exception: Could not find job application_1474295224072_0013.: Traceback (most recent call last):
  File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/handlers/base.py", line 100, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/hue/apps/jobbrowser/src/jobbrowser/views.py", line 62, in decorate
    raise PopupException(_('Could not find job %s.') % jobid, detail=e)
PopupException: Could not find job application_1474295224072_0013.

When looking at a TEZ job (which is working), I am getting the following logs:

[20/Sep/2016 08:47:21 +0000] access       INFO     10.50.0.59 trifil - "GET /jobbrowser/jobs_ats/application_1474295224072_0005 HTTP/1.1"
[20/Sep/2016 08:47:21 +0000] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[20/Sep/2016 08:47:21 +0000] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[20/Sep/2016 08:47:21 +0000] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[20/Sep/2016 08:47:21 +0000] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 404 response
[20/Sep/2016 08:47:21 +0000] access       INFO     10.50.0.59 trifil - "GET /jobbrowser/jobs_ats/application_1474295224072_0005 HTTP/1.1"
[20/Sep/2016 08:47:21 +0000] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[20/Sep/2016 08:47:21 +0000] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[20/Sep/2016 08:47:21 +0000] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response

Re: Job browser in hue does not work for Kerberos and Map Reduce

Contributor

@ Tristan Fily can you share hue.ini configuration here.

Re: Job browser in hue does not work for Kerberos and Map Reduce

New Contributor

Here is attached the hue.ini (I have blanked server names)

hueini.txt

Re: Job browser in hue does not work for Kerberos and Map Reduce

Expert Contributor

Hi @Tristan Fily,

The handle_other(): error looks like a false positive error in HUE (see https://issues.cloudera.org/browse/HUE-2198) so it might not have anything to do with the problem. Please try to gather more information from more logs, change debug log level etc.

/Best regards, Mats

Re: Job browser in hue does not work for Kerberos and Map Reduce

Contributor

@Tristan Fily, Were you able to resolve this issue? IF yes, was there some misconfiguration?

Re: Job browser in hue does not work for Kerberos and Map Reduce

New Contributor

Unfortunately, we never managed to find a work around, and Hortonworks is not supporting Hue. Our plan is to use a much recent version of Hue.

Don't have an account?
Coming from Hortonworks? Activate your account here