How to avoid sharing spark context between two Zeppelin notebooks?

fernando_lopez — Mon, 11 Dec 2017 22:48:25 GMT

When you use Spark from a Zeppelin notebook, and at the end you isse a context stop,

sc.stop()

it affects the context of other running Zeppeling notebooks, making them fail because of no Spark active context. They seem to be sharing the same Spark context.

How can this be avoided?

Re: How to avoid sharing spark context between two Zeppelin notebooks?

mlamairesse — Sun, 18 Aug 2019 01:24:03 GMT

Hi @Fernando Lopez Bello

Sharing of interpreter processes is easily adjustable
Go to the interpreter setting page :

And scroll down to the spark interpreter :

By default interpreters are shared globaly : ie - all notes/users share the same interpreter instance (hence the same spark context)
Change the setting to either "per note" or "per user" depending on your use case :
- Per Note : ie - each note will instantiate a separate interpreter process
- Per User : ie - each user instantiates a seperate interpreter process (which is shared amongst the notes for which he/she has ownership)

Below an article written by one of the original developpers of zeppelin describing interpreter modes :

https://medium.com/@leemoonsoo/apache-zeppelin-interpreter-mode-explained-bae0525d0555

Zeppelin documentation:
https://zeppelin.apache.org/docs/latest/manual/interpreters.html#interpreter-binding-mode

question How to avoid sharing spark context between two Zeppelin notebooks? in Support Questions

How to avoid sharing spark context between two Zeppelin notebooks?

Re: How to avoid sharing spark context between two Zeppelin notebooks?