Livy HTTP 403 Error

[#1]

I am attempting to use the Livy interpreter in Zeppelin in a Kerberized cluster running HDP 2.5.

I am seeing Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 403 Forbidden in the UI, but don't see additional information in the zeppelin or livy logs.

This seems to be a SPNEGO authentication issue of some kind. I tried using curl to connect from the Zeppelin node and was able to authenticate using --negotiate with the ticket in the cache.

How can I troubleshoot this error further?

[#2]

@slachterman

Not sure if I have enough information but couple of things come to mind. In a Kerberized environment, you need to do a kinit and then use proxy user. How are you doing that? I guess you already know this but you cannot for example use keytab and proxy user together. See the details below:

https://issues.cloudera.org/browse/LIVY-98

Also, I have not used livy with zeppelin but according to the docs, you should use "livy.spark" (May be this is just another way of doing it but I thought I'll point out).

https://zeppelin.apache.org/docs/0.6.0/interpreter/livy.html

[#3]

Thanks @mqureshi, I was testing with impersonation turned off for the Livy interpreter. The curl test was just to confirm that the zeppelin service could authenticate to the Livy REST API using SPNEGO. My assumption with impersonation turned off is that Livy would launch the Spark application as the livy principal.

Interestingly, with impersonation enabled, I am seeing a different error: java.net.ConnectException: Connection refused (Connection refused).

By the way, behavior is the same with interpreter specified as 'livy.spark'.

[#4 · Accepted solution]

I had the same issue as described here: https://community.hortonworks.com/questions/69697/getting-error-user-session-not-found-403-when-usin...

Changing livy.superusers in the Custom Livy conf in Spark configuration so that the cluster name is in lowercase allowed that first 403 error to go away. I am now running into another issue where the error in the UI is "Cannot start spark" and in the logs there appears to be an issue authenticating to the Hive metastore using Kerberos.

This may be https://issues.apache.org/jira/browse/SPARK-13478 for Spark 1.6.2 and Zeppelin 0.6.0, I'm researching further.

[#5]

Right, spark of HDP fix this issue as we backport this to HDP 2.5

[#6]

@jzhang I am seeing this issue in an HDP 2.5 cluster (Zeppelin 0.6.0 and Spark 1.6.2). In which HDP release was the fix backported?

[#7]

The hive metastore security issue should be fixed in HDP 2.5. (Please use yarn-cluster mode, yarn-client mode still has this issue) Could you attach the logs and configs ?

[#8]

@jzhang good call, I changed to yarn-cluster mode for the Livy interpreter and was not able to reproduce the error in HDP 2.5.