question Re: Pyspark stuck at Stage 0 in Archives of Support Questions (Read Only)

Pyspark stuck at Stage 0

hawkphil — Fri, 16 Sep 2022 09:52:08 GMT

Hi all

I installed Cloudera 5.5 and Spark YARN. I uploaded a small file as below:

Then I ran pyspark as hdfs user and did a simple exercise but it got stuck at Stage 0 as screenshot:

It never returned anything. Can someone point me out a way to troubleshoot and fix this?

Re: Pyspark stuck at Stage 0

Harsh J — Sat, 12 Dec 2015 18:45:12 GMT

What is your NodeManager configuration's yarn.nodemanager.resource.memory-mb value set to?

Its possible that YARN is unable to allocate a container for the executors, due to too low value of that configuration, in which case things could hang this way. You could raise that config by another 1 GB and restart the cluster/re-run the shell to see if it resolves the issue.

You can also check the Spark AM's log (visit your RM Web UI and click through the RUNNING Spark application, and click on the "logs" link for its Application Master). It may show what it is stuck on, if its yet to spawn up an executor, or if its something else.

Re: Pyspark stuck at Stage 0

hawkphil — Mon, 14 Dec 2015 17:20:31 GMT

I found out my YARN deployment got messed up because I didn't add more NodeManager in after I added new hosts manually. Ops!