Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to restart sparkcontext for certain user?

How to restart sparkcontext for certain user?

New Contributor

Hello!

My situation is following. We have several users running their PySpark tasks via Zeppelin. Spark interpreter is set to "per user, isolated" mode, so each user has its own sparkContext when launch first pyspark paragraph.

The problem occurs when certain user's task get stuck, and there is a need to kill its sparkContext and rerun zeppelin notebook. As admin, I can kill application in YARN UI. However, notebook continues to "run" nonexistend task in paragraph. So far, I found the only way - restart zeppelin from Ambari. But I do not want to interrupt the work process of other users.

So, is there any way to restart notebook and sparkContext for a particular user ?

Thank you in advance.

8 REPLIES 8

Re: How to restart sparkcontext for certain user?

Cloudera Employee

@Stanislav NikitinYou can restart the Spark Interpreter, instead of restarting zeppelin from Ambari.

Re: How to restart sparkcontext for certain user?

New Contributor

Thank you for answering.

I did this, but in this case interpreter is restarted for all users, which drops their progress, I checked.

Re: How to restart sparkcontext for certain user?

@Stanislav Lysikov

To stop the sparkContext user can do a sc.stop(). However if your task is stuck I don 't think you will be able to execute this statement. The only way i see is to kill the yarn application if you cannot restart the interpreter.

Re: How to restart sparkcontext for certain user?

New Contributor

Restarting the interpreter will affect on all users

Re: How to restart sparkcontext for certain user?

Right, As mentioned above the only option i see is to kill the yarn application.

Re: How to restart sparkcontext for certain user?

New Contributor

So, if there is really no way to restart certain sparkContext, what is the reason developers of Zeppelin do not add this possibility? I know it is possible using Jupyter Notebook.

Re: How to restart sparkcontext for certain user?

Community Manager

The above question and the entire reply thread below was originally posted in the Community Help track. On Wed Jun 26 21:24 UTC 2019, a member of the HCC moderation staff moved it to the Data Science & Advanced Analytics track. The Community Help Track is intended for questions about using the HCC site itself, not technical questions about Zeppelin, PySpark or the YARN UI.

Bill Brooks, Community Manager
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Re: How to restart sparkcontext for certain user?

New Contributor

You can reset the interpreter at the note level. If you go to the note interpreter bindings there is the ability to reset each individual interpreter individually.