Can't connect to Livy through Kerberos

[#1]

I'm working with Kerberized HDP 2.6 cluster with Livy2 service, talking to Spark LLAP.

Under any of the host server I'm able to successfully connect to Livy i.e. through curl:

curl --negotiate -u : host-with-livy.com:8998/sessions

Question: how to connect to livy service from other instances, which are not in the cluster?

For example, I'm trying to connect from a dockerized ubuntu instance, sitting in one of the host machines (so it's able to connect to any of the machines, but can have a different hostname set, i.e. dockerized-instance.host-with-livy.com). What I've tried:

1. installed the kerberos client, copied the krb5.conf file from the server.
2. created krbtgt/...@REALM.COM, HTTP/...@REALM.COM principles in the Kerberos server, created their keytabs
3. kinit is successful, trying to connect to livy through curl does create a second ticket for HTTP/...@REALM when checking through klist.
4. (is this enough, or am I missing some crucial steps?)

However, connecting to livy draws an error:

error 403: org.apache.hadoop.security.authentication.client.AuthenticationException

I've noticed in the livy2-conf file that livy.server.auth.kerberos.principal=HTTP/_HOST@REALM.COM -- if I understand correctly, my guess is that only the _hosts from the cluster will be able to authenticate? If so, is it possible to specify additional connection settings, allowing connections from external instances, such as the mentioned dockerized instance?

Second question: Am I missing some steps while configuring the kerberos client? Since setting livy.server.auth.kerberos.principal=HTTP/...@REALM to match the hostname of the dockerized instance and replacing the appropriate keytabs in livy.server.auth.kerberos.keytab setting, theconnection still fails, suggesting that I'm doing something wrong.

Any help would be appreciated!

[#2]

HI @Javert Kirilov,

I was facing this issue when trying accessing livy with Python scripts. Please try something like this , if curl is blocking you.

You may need to install python's requests package.

import json, pprint, requests, textwrap
from requests_kerberos import HTTPKerberosAuth
host='http://LIVY_HOST:LIVY_PORT'
data = {'kind': 'spark'}
headers = {'Requested-By': 'MY_USER_ID','Content-Type': 'application/json'}
auth=HTTPKerberosAuth()
r0 = requests.post(host + '/sessions', data=json.dumps(data), headers=headers,auth=auth)
r0.json()

Regards,

SS

[#3]

Thanks, @Smart Solutions, will try!

By the way, how did you configure the client's Kerberos keytabs? Is copying the original keytab of HTTP/... to both hosts is enough? Since I have no experience working with Kerberos, I have a hunch that my problems are due to some faulty configurations, however, I can't find a way proper to debug it (i.e. to see what's happening in the server and what's not working).

[#4]

Using

auth=HTTPKerberosAuth()

will pass your Kerberos ticket in my understanding. It is similar to --negotiate, in curl.