Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark 1.4 availability

Spark 1.4 availability

Rising Star

Hi dear experts!

 

could anybodu know when Spark 1.4 will be available in CDH?

 

thaks!

3 REPLIES 3
Highlighted

Re: Spark 1.4 availability

Master Collaborator

You can run it 'manually' as a YARN app in CDH 5.4. It should largely Just Work.

 

As you can make out here, CDH 5.5 is actually already based on what will be Spark 1.5:

https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.10/1.5.0-cd...

 

CDH 5.4 can't update a minor release to 1.4 in a maintenance release, but 5.5 looks like it will jump to 1.5.

Highlighted

Re: Spark 1.4 availability

Rising Star

Thank you so much!

and could you tell is there any available some instruction for running 1.4 manyally on YARN?

thanks!

Highlighted

Re: Spark 1.4 availability

Super Collaborator

We do not have any instructions available since we do not support it. Spark has a Hadoop, among others, dependency and it needs to be compiled against the correct version of Hadoop.

You would need to create your own Spark install: compile Spark against the CDH 5.4 artifacts and make sure you reference the correct dependencies.

 

Wilfred

Don't have an account?
Coming from Hortonworks? Activate your account here