question Re: Which CDH release will include Spark 1.4.x? in Support Questions

Which CDH release will include Spark 1.4.x?

TS — Fri, 16 Sep 2022 09:33:11 GMT

Does anyone which CDH release will include/support Spark 1.4?

If so, any timetable?

Re: Which CDH release will include Spark 1.4.x?

srowen — Fri, 03 Jul 2015 12:31:35 GMT

Presumably CDH 5.5, since a new minor release is needed to update a minor release of a component in general. There aren't timeframes for this, but you can see CDH is typically on a 4-6 month minor release cycle and 5.4 was out 2 months ago.

Re: Which CDH release will include Spark 1.4.x?

srowen — Fri, 03 Jul 2015 12:34:03 GMT

PS I should say too that you should be able to use 1.4 with CDH 5.4 and have it generally work; this requires a little bit of understanding of how to get a build on a machine and run from that build, but otherwise it's a YARN app and modulo some dependency issues at the edge maybe, should just work.

Re: Which CDH release will include Spark 1.4.x?

TS — Fri, 03 Jul 2015 12:34:29 GMT

Greta, thank you for your quick response!

Hoping to have CDH 5.5 released sooner than 4-5 months 🙂

Re: Which CDH release will include Spark 1.4.x?

TS — Fri, 03 Jul 2015 12:40:13 GMT

That's even better!

I could give a try, at least for my Lab environment.

Could you please provide some info, links, docs, blogs how this could be done?

Re: Which CDH release will include Spark 1.4.x?

srowen — Fri, 03 Jul 2015 12:46:42 GMT

If it were me, I'd download the source for 1.4.0 and build for the exact CDH artifacts to be safest. See http://spark.apache.org/docs/latest/hadoop-third-party-distributions.html Then just try running the local copy of bin/spark-shell etc from that distribution. You need to use YARN masters. I won't 100% guarantee that works but see no reason it wouldn't. The build flags are probably like -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.3 -Pyarn

Re: Which CDH release will include Spark 1.4.x?

cerd — Wed, 23 Sep 2015 15:52:16 GMT

I had tried this but seemed to have some trouble in using things like pyspark, etc. - is there a gist or something somewhere with exact steps for CDH?

I will try again and post what I did.