How to restart sparkcontext for certain user?

[#1]

Hello!

My situation is following. We have several users running their PySpark tasks via Zeppelin. Spark interpreter is set to "per user, isolated" mode, so each user has its own sparkContext when launch first pyspark paragraph.

The problem occurs when certain user's task get stuck, and there is a need to kill its sparkContext and rerun zeppelin notebook. As admin, I can kill application in YARN UI. However, notebook continues to "run" nonexistend task in paragraph. So far, I found the only way - restart zeppelin from Ambari. But I do not want to interrupt the work process of other users.

So, is there any way to restart notebook and sparkContext for a particular user ?

Thank you in advance.

[#2]

@Stanislav NikitinYou can restart the Spark Interpreter, instead of restarting zeppelin from Ambari.

[#3]

Thank you for answering.

I did this, but in this case interpreter is restarted for all users, which drops their progress, I checked.

[#4]

@Stanislav Lysikov

To stop the sparkContext user can do a sc.stop(). However if your task is stuck I don 't think you will be able to execute this statement. The only way i see is to kill the yarn application if you cannot restart the interpreter.

[#5]

Restarting the interpreter will affect on all users

[#6]

Right, As mentioned above the only option i see is to kill the yarn application.

[#7]

So, if there is really no way to restart certain sparkContext, what is the reason developers of Zeppelin do not add this possibility? I know it is possible using Jupyter Notebook.

[#8]

The above question and the entire reply thread below was originally posted in the Community Help track. On Wed Jun 26 21:24 UTC 2019, a member of the HCC moderation staff moved it to the Data Science & Advanced Analytics track. The Community Help Track is intended for questions about using the HCC site itself, not technical questions about Zeppelin, PySpark or the YARN UI.

[#9]

You can reset the interpreter at the note level. If you go to the note interpreter bindings there is the ability to reset each individual interpreter individually.