Support Questions
Find answers, ask questions, and share your expertise

SAP HANA VORA fails with SPARK 1.5 CDH 5.5

SAP HANA VORA fails with SPARK 1.5 CDH 5.5

When I try to create a SapSQLContext in spark-shell I get following error:

 

 

scala> import org.apache.spark.sql.SapSQLContext
import org.apache.spark.sql.SapSQLContext

scala> val vc = new SapSQLContext(sc)
16/02/12 15:13:55 ERROR SapSQLContext: Spark Version mismatch: Supported: 1.5.0,1.5.1,1.5.2, Runtime is: 1.5.0-cdh5.5.1
java.lang.RuntimeException: Termination due to Spark version mismatch
        at org.apache.spark.sql.CommonSapSQLContext$class.checkSparkVersion(CommonSapSQLContext.scala:36)
        at org.apache.spark.sql.SapSQLContext.checkSparkVersion(SapSQLContext.scala:23)
        at org.apache.spark.sql.CommonSapSQLContext$class.$init$(CommonSapSQLContext.scala:19)

 

Is there a work around for this error?

5 REPLIES 5
Highlighted

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

Master Collaborator
You would have to ask SAP to adjust their version-checking logic.
Highlighted

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

I think it is a bug in the CDH SPARK. In CDH 5.4 the SPARK version was:
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.3.0
/_/

In CDH 5.5 the SPARK version is
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.5.0-cdh5.5.1
/_/
Why does the cdh suffix appear now in the version?
Highlighted

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

Master Collaborator
No. The Spark version has always been "x.y.x-cdha.b.c". The banner
text is irrelevant.

The SAP code is looking for "x.y.z" only to confirm it's a Spark 1.5
distro, but this is incorrect, since it's rejecting a version it
should accept. It simply needs to ignore suffixes. This is pretty
standard practice for semantic versioning.

Highlighted

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

The Host Inspector showing the SPARK version as 1.5.0+cdh5.5.1+94
So in my understanding the version should be 1.5.0 as it was in 1.3.0 in CDH5.4
scala> sc.version
res0: String = 1.3.0

and not

scala> sc.version

res0: String = 1.5.0-cdh5.5.1
Highlighted

Re: SAP HANA VORA fails with SPARK 1.5 CDH 5.5

Master Collaborator
Oops, it really should have always been 1.3.0-cdhx.x.x since that is
the version in the Maven artifacts.
In any event, 1.5.0-cdh5.5.1 is a valid version string that this third
party library should not reject.
Don't have an account?