I have this problem too. No word from Cloudera if and when they will ship Spark 2 RPM packages for CDH 5. I think you could install Spark 2 from Apache Bigtop (or build your own RPM) on an edge node and deploy Spark 2 jobs with Yarn. With Yarn you would not need Spark Worker packages on the worker nodes. Edit: I just tried this with Apache Zeppelin and it seem to work. I took the tar.gz from spark.apache.org and extracted it on an edge node. Then configured zeppelin-env.sh with the following variables: export HADOOP_USER_NAME=spark
When I run spark code in Zeppelin I can see that they get executed with Yarn. They can access HDFS files.
... View more