Reply
Explorer
Posts: 14
Registered: ‎06-13-2016

Spark 2.0 / livy server 3

Hi.

Downloaded spark 2.0 from http://www.cloudera.com/downloads/spark2/2-0.html and deployed it according to the instructions for cloudera manager.

 

Installed latest livy server from github with options "mvn clean package -DskipTests -Dspark-2.0 -Dscala-2.11".

 

During startup livy server can't figure out the spark-submit version due to a regex problem in livy server LivySparkUtils.scala file.

 

"Fail to parse Spark version from 2.0.0.cloudera1"

 

Here is a similar issue, https://issues.cloudera.org/browse/LIVY-229.

 

Is there any official guidelines from Cloudera on what version of Livy server to use for Spark 2.0?

 

And will livy server be packaged in future CDH releases?

Explorer
Posts: 6
Registered: ‎01-09-2017

Re: Spark 2.0 / livy server 3

Hi,

 

you could replace the last '\d' in the regex with a dot and livy will start corectly:

 

 

  def formatSparkVersion(version: String): (Int, Int) = {
    val versionPattern = """(\d)+\.(\d)+(?:[\.-]\d*)*""".r

fixed:

  def formatSparkVersion(version: String): (Int, Int) = {
    val versionPattern = """(\d)+\.(\d)+(?:[\.-].*)*""".r

 

 

Highlighted
Explorer
Posts: 14
Registered: ‎06-13-2016

Re: Spark 2.0 / livy server 3

Thanks! 

Announcements