Failing to kill a job from Job Browser on kerberized cluster

[#1]

I'm using CDH 5.5.0 with kerberos and Sentry enabled. Trying to kill a job from the job browser fails with the message "There was a problem communicating with the server: The default static user cannot carry out this operation. (error 403)"

I can kill the same job using the yarn application -kill command.

I guess this is a configuration issue. Could someone assist me in getting this right so that I can can kill jobs from the Job Browser?

Stack trace:

[26/Jan/2016 10:15:30 +0100] access       WARNING  10.128.42.143 di23060584 - "POST /jobbrowser/jobs/application_1453476679853_0011/kill HTTP/1.1"
[26/Jan/2016 10:15:30 +0100] connectionpool INFO     Resetting dropped connection: ip-10-255-2-7.eu-west-1.compute.internal
[26/Jan/2016 10:15:30 +0100] kerberos_    ERROR    handle_mutual_auth(): Mutual authentication unavailable on 403 response
[26/Jan/2016 10:15:30 +0100] views        ERROR    Killing job
Traceback (most recent call last):
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/apps/jobbrowser/src/jobbrowser/views.py", line 246, in kill_job
    job.kill()
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/apps/jobbrowser/src/jobbrowser/yarn_models.py", line 185, in kill
    return self.api.kill(self.id)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/mapreduce_api.py", line 117, in kill
    get_resource_manager(self._user).kill(app_id) # We need to call the RM
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py", line 124, in kill
    return self._execute(self._root.put, 'cluster/apps/%(app_id)s/state' % {'app_id': app_id}, params=params, data=json.dumps(data), contenttype=_JSON_CONTENT_TYPE)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/libs/hadoop/src/hadoop/yarn/resource_manager_api.py", line 141, in _execute
    response = function(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 136, in put
    return self.invoke("PUT", relpath, params, data, self._make_headers(contenttype))
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/core/src/desktop/lib/rest/resource.py", line 78, in invoke
    urlencode=self._urlencode)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/desktop/core/src/desktop/lib/rest/http_client.py", line 161, in execute
    raise self._exc_class(ex)
RestException: The default static user cannot carry out this operation. (error 403)
[26/Jan/2016 10:15:30 +0100] middleware   INFO     Processing exception: The default static user cannot carry out this operation. (error 403): Traceback (most recent call last):
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/build/env/lib/python2.6/site-packages/Django-1.6.10-py2.6.egg/django/db/transaction.py", line 371, in inner
    return func(*args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/apps/jobbrowser/src/jobbrowser/views.py", line 83, in decorate
    return view_func(request, *args, **kwargs)
  File "/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hue/apps/jobbrowser/src/jobbrowser/views.py", line 249, in kill_job
    raise PopupException(e)
PopupException: The default static user cannot carry out this operation. (error 403)

[#2]

You are very probably hitting this bug if you are on 5.5.1:
https://review.cloudera.org/r/6769/

(will be fixed in 5.5.3)

[#3]

Thank you for your reply!

I also noticed that if I check the option "Enable Kerberos Authentication for HTTP Web-Consoles" in the YARN configuration I can make the kill button work. However, this will enable kerberos for web pages such as for the History Server and Resouce Manager. However, we do not want kerberos authentication on these pages.

So, with the fix in CDH5.5.3 the kill button will work without enabeling the above option I assume?

[#4 · Accepted solution]

AFAIK starting in 5.5 you need to enable kerberos on the Web UI in any
case, as a change in Yarn now requires it for kill/starting actions with
Kerberos.

The above caching issue might prevent the kill button to work in certain
cases until 5.5.3: https://issues.cloudera.org/browse/HUE-3107

So to sum-up, maybe you were lucky if you were testing it on 5.5.1, as all
should only work properly in 5.5.3 + checking the spnego for the Web UI
(Yarn guys you fix their get delegation toekn API at some point instead)

Romain