Member since
02-05-2021
2
Posts
0
Kudos Received
0
Solutions
03-16-2021
06:17 AM
Hey sorry for the delay in getting back to you. You don't need to have spark installed on the CDSW nodes, but you DO need to have the Spark Gateway roles installed. This is how CDSW picks up the configuration for Spark and knows which jar files to use, etc. If you have this working with Python, does that mean you can run the "pi.py" example from a Python session which uses Spark? If so then the problem is not the gateways. If spark is not installed then yes I think this is probably the problem since the Scala workbench tries to create a SparkSession object immediately. This might be why your stack trace has lots of spark paths in it near the end.
... View more