So I found a solution for getting it to work on 5.8.3 and 5.9. I added the following configuration to the workflow: <spark-opts>--conf spark.yarn.jar=local:/opt/cloudera/parcels/CDH/lib/spark/lib/spark-assembly.jar</spark-opts> I don't know why that is necessary on 5.8.3 and 5.9 but not on 5.8.0.
... View more
