Created 08-08-2018 02:05 PM
Hello!
My situation is following. We have several users running their PySpark tasks via Zeppelin. Spark interpreter is set to "per user, isolated" mode, so each user has its own sparkContext when launch first pyspark paragraph.
The problem occurs when certain user's task get stuck, and there is a need to kill its sparkContext and rerun zeppelin notebook. As admin, I can kill application in YARN UI. However, notebook continues to "run" nonexistend task in paragraph. So far, I found the only way - restart zeppelin from Ambari. But I do not want to interrupt the work process of other users.
So, is there any way to restart notebook and sparkContext for a particular user ?
Thank you in advance.
Created 08-08-2018 03:04 PM
@Stanislav NikitinYou can restart the Spark Interpreter, instead of restarting zeppelin from Ambari.
Created 08-08-2018 03:21 PM
Thank you for answering.
I did this, but in this case interpreter is restarted for all users, which drops their progress, I checked.
Created 08-08-2018 05:32 PM
To stop the sparkContext user can do a sc.stop(). However if your task is stuck I don 't think you will be able to execute this statement. The only way i see is to kill the yarn application if you cannot restart the interpreter.
Created 08-09-2018 09:13 AM
Restarting the interpreter will affect on all users
Created 08-09-2018 09:41 AM
Right, As mentioned above the only option i see is to kill the yarn application.
Created 08-10-2018 06:55 AM
So, if there is really no way to restart certain sparkContext, what is the reason developers of Zeppelin do not add this possibility? I know it is possible using Jupyter Notebook.
Created 06-26-2019 09:26 PM
The above question and the entire reply thread below was originally posted in the Community Help track. On Wed Jun 26 21:24 UTC 2019, a member of the HCC moderation staff moved it to the Data Science & Advanced Analytics track. The Community Help Track is intended for questions about using the HCC site itself, not technical questions about Zeppelin, PySpark or the YARN UI.
Created 06-27-2019 12:49 AM
You can reset the interpreter at the note level. If you go to the note interpreter bindings there is the ability to reset each individual interpreter individually.