Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SAP HANA VORA fails with SPARK 1.5 CDH 5.5

SAP HANA VORA fails with SPARK 1.5 CDH 5.5

When I try to create a SapSQLContext in spark-shell I get following error:

 

 

scala> import org.apache.spark.sql.SapSQLContext
import org.apache.spark.sql.SapSQLContext

scala> val vc = new SapSQLContext(sc)
16/02/12 15:13:55 ERROR SapSQLContext: Spark Version mismatch: Supported: 1.5.0,1.5.1,1.5.2, Runtime is: 1.5.0-cdh5.5.1
java.lang.RuntimeException: Termination due to Spark version mismatch
        at org.apache.spark.sql.CommonSapSQLContext$class.checkSparkVersion(CommonSapSQLContext.scala:36)
        at org.apache.spark.sql.SapSQLContext.checkSparkVersion(SapSQLContext.scala:23)
        at org.apache.spark.sql.CommonSapSQLContext$class.$init$(CommonSapSQLContext.scala:19)

 

Is there a work around for this error?

5 REPLIES 5

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

Master Collaborator
You would have to ask SAP to adjust their version-checking logic.

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

I think it is a bug in the CDH SPARK. In CDH 5.4 the SPARK version was:
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.3.0
/_/

In CDH 5.5 the SPARK version is
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.0-cdh5.5.1
/_/
Why does the cdh suffix appear now in the version?

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

Master Collaborator
No. The Spark version has always been "x.y.x-cdha.b.c". The banner
text is irrelevant.

The SAP code is looking for "x.y.z" only to confirm it's a Spark 1.5
distro, but this is incorrect, since it's rejecting a version it
should accept. It simply needs to ignore suffixes. This is pretty
standard practice for semantic versioning.

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

The Host Inspector showing the SPARK version as 1.5.0+cdh5.5.1+94
So in my understanding the version should be 1.5.0 as it was in 1.3.0 in CDH5.4
scala> sc.version
res0: String = 1.3.0

and not

scala> sc.version

res0: String = 1.5.0-cdh5.5.1
Highlighted

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

Master Collaborator
Oops, it really should have always been 1.3.0-cdhx.x.x since that is
the version in the Maven artifacts.
In any event, 1.5.0-cdh5.5.1 is a valid version string that this third
party library should not reject.