Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oryx 2 dependencies

Oryx 2 dependencies

Explorer

Sean,

 

Spark is the platform used by Oryx 2.

Due to this dependency, how Oryx 2 is releasing with Spark ?

 

For example, Oryx 2 uses SparkML-ALS and there is reported issue with Spark1.3

(https://groups.google.com/a/cloudera.org/forum/#!topic/oryx-user/_jfv6t00o2I).

It's indicated there was a fix for Spark1.4.

Can we just get Oryx 2 and compile with Spark 1.4 ?

Or, if CDH has updated with Spark 1.4 and it will just work with Oryx 2 we installed (as long as the SparkML-ALS interfaces not changed).

It looks the latest CDH5.4.5 still uses Spark 1.3 which indicates there is no resolution posted for Spark 1.4.

Any suggestion to handle these dependencies issue related to Oryx 2?

 

Jason

1 REPLY 1
Highlighted

Re: Oryx 2 dependencies

Master Collaborator
It's compiled with Spark and Hadoop dependencies "provided", so they
must be available from the cluster. You should be able to use later
versions of Spark and Hadoop, as long as they're backwards-compatible,
yes. Spark 1.3+ is required.

If CDH ships a later Spark, that's fine then. There is generally no
need to recompile for the later version.

CDH 5.4 = 1.3 and can't upgrade a minor release in a minor release.
CDH 5.5 = 1.5. You can run Spark 1.4 manually on CDH 5.4, which takes
some care, or wait for 5.5. Yes, it is a Spark problem.

I don't see a dependencies issue?

Don't have an account?
Coming from Hortonworks? Activate your account here