Pyspark stuck at Stage 0

[#1]

Hi all

I installed Cloudera 5.5 and Spark YARN. I uploaded a small file as below:

Screen Shot 2015-12-09 at 10.27.23 AM.png

Then I ran pyspark as hdfs user and did a simple exercise but it got stuck at Stage 0 as screenshot:

Screen Shot 2015-12-09 at 10.30.39 AM.png

It never returned anything. Can someone point me out a way to troubleshoot and fix this? 

[#2 · Accepted solution]

What is your NodeManager configuration's yarn.nodemanager.resource.memory-mb value set to?

Its possible that YARN is unable to allocate a container for the executors, due to too low value of that configuration, in which case things could hang this way. You could raise that config by another 1 GB and restart the cluster/re-run the shell to see if it resolves the issue.

You can also check the Spark AM's log (visit your RM Web UI and click through the RUNNING Spark application, and click on the "logs" link for its Application Master). It may show what it is stuck on, if its yet to spawn up an executor, or if its something else.

[#3]

I found out my YARN deployment got messed up because I didn't add more NodeManager in after I added new hosts manually. Ops!