Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache Zeppelin Livy can't initialize SparkContext on YARN mode

Highlighted

Apache Zeppelin Livy can't initialize SparkContext on YARN mode

New Contributor
livy-stderr.txt

Hey there,

I'm trying to use Spark2 with Livy in Apache Zeppelin. But sadly I can get it to work, because the Livy session always shuts down with the error that it can't initialize the SparkContext. I can't seem to find the problem.

The YARN resource manager doesn't even gets a request to start an application (as far as I can see in the logs), so I don't think the problem lies there. Also all other applications work just fine with YARN, including Spark submit and shell requests.

Here is the livy.conf:

livy.environment production
livy.impersonation.enabled true
livy.repl.enableHiveContext true
livy.rsc.rpc.server.address 192.168.1.22
livy.server.csrf_protection.enabled true
livy.server.port 8999
livy.server.recovery.mode recovery
livy.server.recovery.state-store filesystem
livy.server.recovery.state-store.url /livy2-recovery
livy.server.session.timeout 3600000
livy.spark.master yarn
livy.superusers zeppelin, livy

I included the stderr log for the failed attempt.

I would appreciate any help with this problem.

Best regards,

Markus

2 REPLIES 2

Re: Apache Zeppelin Livy can't initialize SparkContext on YARN mode

New Contributor

I could see the following in the error log:

ERROR: org.apache.hadoop.security.authorize.AuthorizationException: User: livy is not allowed to impersonate admin

Looks like your hadoop cluster is a secure one. You need to grant livy the ability to impersonate as the originating user. You need to add two properties to core-site.xml. Take a look at this guide.

Re: Apache Zeppelin Livy can't initialize SparkContext on YARN mode

New Contributor

Can you please let us know which spark and zeppelin version you are using?

Is Kerberos is enabled in you cluster ?


Below are few things to check to resolve this issue :

1)Make sure you configured livy.host and livy.group in custom-core.site (hdfs->Custom-core.site)

Make sure they are configure to * for livy


2)Also you need to configure livy.superuser in your Livy interpreter too.


3)Yarn Queue access ?


4)if ranger is enabled , please make sure you have access to YARN queqe


Hope this helps