07-03-2015 05:31 AM
Presumably CDH 5.5, since a new minor release is needed to update a minor release of a component in general. There aren't timeframes for this, but you can see CDH is typically on a 4-6 month minor release cycle and 5.4 was out 2 months ago.
07-03-2015 05:34 AM
PS I should say too that you should be able to use 1.4 with CDH 5.4 and have it generally work; this requires a little bit of understanding of how to get a build on a machine and run from that build, but otherwise it's a YARN app and modulo some dependency issues at the edge maybe, should just work.
07-03-2015 05:46 AM
If it were me, I'd download the source for 1.4.0 and build for the exact CDH artifacts to be safest. See http://spark.apache.org/docs/latest/hadoop-third-party-distributions.html Then just try running the local copy of bin/spark-shell etc from that distribution. You need to use YARN masters. I won't 100% guarantee that works but see no reason it wouldn't. The build flags are probably like -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.3 -Pyarn
09-23-2015 08:52 AM
I had tried this but seemed to have some trouble in using things like pyspark, etc. - is there a gist or something somewhere with exact steps for CDH?
I will try again and post what I did.