Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to avoid sharing spark context between two Zeppelin notebooks?

Solved Go to solution
Highlighted

How to avoid sharing spark context between two Zeppelin notebooks?

Rising Star

When you use Spark from a Zeppelin notebook, and at the end you isse a context stop,

sc.stop()

it affects the context of other running Zeppeling notebooks, making them fail because of no Spark active context. They seem to be sharing the same Spark context.

How can this be avoided?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to avoid sharing spark context between two Zeppelin notebooks?

Cloudera Employee

Hi @Fernando Lopez Bello

Sharing of interpreter processes is easily adjustable
Go to the interpreter setting page :

45388-screen-shot-2017-12-12-at-121129.png

And scroll down to the spark interpreter :

45391-screen-shot-2017-12-12-at-121346.png

By default interpreters are shared globaly : ie - all notes/users share the same interpreter instance (hence the same spark context)
Change the setting to either "per note" or "per user" depending on your use case :
- Per Note : ie - each note will instantiate a separate interpreter process
- Per User : ie - each user instantiates a seperate interpreter process (which is shared amongst the notes for which he/she has ownership)

Below an article written by one of the original developpers of zeppelin describing interpreter modes :

https://medium.com/@leemoonsoo/apache-zeppelin-interpreter-mode-explained-bae0525d0555

Zeppelin documentation:
https://zeppelin.apache.org/docs/latest/manual/interpreters.html#interpreter-binding-mode

View solution in original post

1 REPLY 1

Re: How to avoid sharing spark context between two Zeppelin notebooks?

Cloudera Employee

Hi @Fernando Lopez Bello

Sharing of interpreter processes is easily adjustable
Go to the interpreter setting page :

45388-screen-shot-2017-12-12-at-121129.png

And scroll down to the spark interpreter :

45391-screen-shot-2017-12-12-at-121346.png

By default interpreters are shared globaly : ie - all notes/users share the same interpreter instance (hence the same spark context)
Change the setting to either "per note" or "per user" depending on your use case :
- Per Note : ie - each note will instantiate a separate interpreter process
- Per User : ie - each user instantiates a seperate interpreter process (which is shared amongst the notes for which he/she has ownership)

Below an article written by one of the original developpers of zeppelin describing interpreter modes :

https://medium.com/@leemoonsoo/apache-zeppelin-interpreter-mode-explained-bae0525d0555

Zeppelin documentation:
https://zeppelin.apache.org/docs/latest/manual/interpreters.html#interpreter-binding-mode

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here