Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark 2.0 / livy server 3

Spark 2.0 / livy server 3

Hi.

Downloaded spark 2.0 from http://www.cloudera.com/downloads/spark2/2-0.html and deployed it according to the instructions for cloudera manager.

 

Installed latest livy server from github with options "mvn clean package -DskipTests -Dspark-2.0 -Dscala-2.11".

 

During startup livy server can't figure out the spark-submit version due to a regex problem in livy server LivySparkUtils.scala file.

 

"Fail to parse Spark version from 2.0.0.cloudera1"

 

Here is a similar issue, https://issues.cloudera.org/browse/LIVY-229.

 

Is there any official guidelines from Cloudera on what version of Livy server to use for Spark 2.0?

 

And will livy server be packaged in future CDH releases?

2 REPLIES 2
Highlighted

Re: Spark 2.0 / livy server 3

Explorer

Hi,

 

you could replace the last '\d' in the regex with a dot and livy will start corectly:

 

 

  def formatSparkVersion(version: String): (Int, Int) = {
    val versionPattern = """(\d)+\.(\d)+(?:[\.-]\d*)*""".r

fixed:

  def formatSparkVersion(version: String): (Int, Int) = {
    val versionPattern = """(\d)+\.(\d)+(?:[\.-].*)*""".r

 

 

Re: Spark 2.0 / livy server 3

Thanks! 

Don't have an account?
Coming from Hortonworks? Activate your account here