Apache Zeppelin Livy can't initialize SparkContext on YARN mode

[#1]

livy-stderr.txt

Hey there,

I'm trying to use Spark2 with Livy in Apache Zeppelin. But sadly I can get it to work, because the Livy session always shuts down with the error that it can't initialize the SparkContext. I can't seem to find the problem.

The YARN resource manager doesn't even gets a request to start an application (as far as I can see in the logs), so I don't think the problem lies there. Also all other applications work just fine with YARN, including Spark submit and shell requests.

Here is the livy.conf:

livy.environment production
livy.impersonation.enabled true
livy.repl.enableHiveContext true
livy.rsc.rpc.server.address 192.168.1.22
livy.server.csrf_protection.enabled true
livy.server.port 8999
livy.server.recovery.mode recovery
livy.server.recovery.state-store filesystem
livy.server.recovery.state-store.url /livy2-recovery
livy.server.session.timeout 3600000
livy.spark.master yarn
livy.superusers zeppelin, livy

I included the stderr log for the failed attempt.

I would appreciate any help with this problem.

Best regards,

Markus

[#2]

I could see the following in the error log:

ERROR: org.apache.hadoop.security.authorize.AuthorizationException: User: livy is not allowed to impersonate admin

Looks like your hadoop cluster is a secure one. You need to grant livy the ability to impersonate as the originating user. You need to add two properties to core-site.xml. Take a look at this guide.

[#3]

Can you please let us know which spark and zeppelin version you are using?

Is Kerberos is enabled in you cluster ?

Below are few things to check to resolve this issue :

1)Make sure you configured livy.host and livy.group in custom-core.site (hdfs->Custom-core.site)

Make sure they are configure to * for livy

2)Also you need to configure livy.superuser in your Livy interpreter too.

3)Yarn Queue access ?

4)if ranger is enabled , please make sure you have access to YARN queqe

Hope this helps