Reply
Cloudera Employee
Posts: 40
Registered: ‎11-16-2015

Re: Unable to run multiple pyspark sessions

@hedy thanks for sharing.

 

The workaround you received makes sense when you are not using any cluster manager(?)

 

Local mode ( --master local[i]is generally seen if you want to test or debug something quickly since there will be only one JVM launched on the node from where you are running pyspark and this JVM will act as driver, executor, and master -> all-in-one. But of course with local mode, you lose the scalability and resource management that a cluster manager provides. If you want to debug why simultaneous spark shells are not working when using Spark-On-Yarn, we need to diagnose this from YARN perspective (troubleshooting steps shared in the last post). Let us know.

Announcements