Reply
TS
Contributor
Posts: 84
Registered: ‎02-10-2015
Accepted Solution

Which CDH release will include Spark 1.4.x?

Does anyone which CDH release will include/support Spark 1.4?

If so, any timetable?

 

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Which CDH release will include Spark 1.4.x?

Presumably CDH 5.5, since a new minor release is needed to update a minor release of a component in general. There aren't timeframes for this, but you can see CDH is typically on a 4-6 month minor release cycle and 5.4 was out 2 months ago.

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Which CDH release will include Spark 1.4.x?

PS I should say too that you should be able to use 1.4 with CDH 5.4 and have it generally work; this requires a little bit of understanding of how to get a build on a machine and run from that build, but otherwise it's a YARN app and modulo some dependency issues at the edge maybe, should just work.

TS
Contributor
Posts: 84
Registered: ‎02-10-2015

Re: Which CDH release will include Spark 1.4.x?

Greta, thank you for your quick response!

Hoping to have CDH 5.5 released sooner than 4-5 months :-)

 

TS
Contributor
Posts: 84
Registered: ‎02-10-2015

Re: Which CDH release will include Spark 1.4.x?

That's even better!

I could give a try, at least for my Lab environment.

Could you please provide some info, links, docs, blogs how this could be done?

 

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Which CDH release will include Spark 1.4.x?

If it were me, I'd download the source for 1.4.0 and build for the exact CDH artifacts to be safest. See http://spark.apache.org/docs/latest/hadoop-third-party-distributions.html  Then just try running the local copy of bin/spark-shell etc from that distribution. You need to use YARN masters. I won't 100% guarantee that works but see no reason it wouldn't. The build flags are probably like -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.3 -Pyarn

New Contributor
Posts: 1
Registered: ‎09-23-2015

Re: Which CDH release will include Spark 1.4.x?

I had tried this but seemed to have some trouble in using things like pyspark, etc. - is there a gist or something somewhere with exact steps for CDH?

I will try again and post what I did.