I have a kerberized HDP 3.1.4 authenticating against an IPA server (Kerberos/LDAP).
Most of the supported web services are working via Knox (WEBHDFS, HDFSUI, YARNUI, YARNUIV2, SPARKHISTORYUI, OOZIEUI, HDFSUI, etc) , in some cases after fixing or replacing the service definitions with the ones from Knox 1.4.0 (some of ones included with HDP 3 have erros o missing redirects).
Now I trying to expose the Livy Rest API via Knox and have tried both the LIVY service definitions in HDP 3 (it includes 3 versions 0.4.0, 0.4.1, 0.4.2) and with the one from Knox 1.4.0.
A side note is that the service definition from upstream Knox 1.4.0 has a lower version number (0.4.0) but seems to be more recent including many additional rewrite rules and not using the "/v1" hack from the preliminary versions.
The service definition for Knox in the ui.xml topology I'm using is:
<service>
<role>LIVYSERVER</role>
<url>http://hdp3-livy:8999</url>
</service>
I'm using curl for testing and I'm able to initialize the "pyspark" session via Knox (authenticated with admin) to Livy as "knox" user and with the user authenticated to Knox (in this case "admin") as the "proxyUser":
$ curl -k -u admin:${SECRET} --config headers-post.conf --data '{"kind": "pyspark"}' $LIVY_URL/sessions
HTTP/1.1 200 OK
{
"appId": null, "appInfo": {"driverLogUrl": null, "sparkUiUrl": null},
"id": 19, "kind": "pyspark", "log": [],
"owner": "knox", "proxyUser": "admin", "state": "starting"
}
However when in the next step I try to execute some code I get an empty response with code "403 Forbidden":
$ curl -k -u admin:${SECRET} --config livy-headers-post.conf $LIVY_URL/sessions/$SESSIONID/statements -d '{"code":"2 + 2"}'
HTTP/1.1 403 Forbidden
If I do the same from Knox server, authenticated with kerberos keytab as user knox and including "proxyUser": "admin" in the JSON request (as Knox does) , I get the same response in the session setup but the subsequent code statement works as expected and executes the code:
[knox@server ~]$ curl --negotiate -u : --config headers-post.conf $LIVY_URL/sessions/$SESSIONID/statements -d '{"code":"2 + 2"}'
HTTP/1.1 200 OK
{
"code": "2 + 2", "id": 0, "output": null, "progress": 0.0, "state": "waiting"
}
As an extra diagnostic I have made a capture of traffic for both executions (via Knox and directly with knox using kinit + proxyUser) and the authentication headers received by Livy from the Knox server are clearly different with each method:
Authencated as admin with Knox:
POST /sessions/19/statements?doAs=admin HTTP/1.1
X-Forwarded-Proto: https
X-Forwarded-Port: 10443
...
X-Forwarded-Context: /gateway/ui
X-Requested-By: admin
Accept: */*
User-Agent: curl/7.29.0
X-XSRF-HEADER: valid
Content-Type: application/json
Transfer-Encoding: chunked
Host: hdp3-livy:8999
Connection: Keep-Alive
Cookie: hadoop.auth="u=knox&p=knox/hdp3-knox@EXAMPLE.COM&t=kerberos&e=1605153545612s=xD7N1bfFduRqQPQ/qtOkg0OVVs6sXC2C2MnTDlUDrSo="
10
{"code":"2 + 2"}
0
Authenticated with kerberos as knox user (--negotiate):
POST /sessions/18/statements HTTP/1.1
User-Agent: curl/7.29.0
Host: hdp3-dtlk-mn01.dtlk.in.iantel.com.uy:8999
Accept: */*
X-Requested-By: admin
Content-Type: application/json
X-XSRF-HEADER:valid
Content-Length: 16
{"code":"2 + 2"}
@
HTTP/1.1 401 Authentication required
Date: Wed, 11 Nov 2020 19:18:50 GMT
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; HttpOnly
Cache-Control: must-revalidate,no-cache,no-store
...
Content-Length: 353
Server: Jetty(9.3.24.v20180605)
POST /sessions/18/statements HTTP/1.1
Authorization: Negotiate YIIC1AYJKoZIhvcSAQICAQBuggLD
......
@
HTTP/1.1 201 Created
Date: Wed, 11 Nov 2020 19:18:50 GMT
WWW-Authenticate: Negotiate YGoGCSqGSIb....
...
Content-Type: application/json;charset=utf-8
Location: /sessions/18/statements/0
Content-Length: 70
Server: Jetty(9.3.24.v20180605)
{"id":0,"code":"2 + 2","state":"waiting","output":null,"progress":0.0}
Does anybody knows how to fix this and make Livy work with HTTPS and basic authentication via the Apache Knox Gateway server using Kerberos?
