question Re: Type Error when attempting Linear Regression in Archives of Support Questions (Read Only)

Type Error when attempting Linear Regression

vjain — Fri, 11 Dec 2015 12:22:11 GMT

import org.apache.spark.mllib.regression.LinearRegressionWithSGD
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer}
import sqlContext.implicits._


val df = sqlContext.sql("select mnemonic, average, median, stddev from wellbook.curve_statistics")


val indexer = new StringIndexer()
  .setInputCol("mnemonic")
  .setOutputCol("mnemonicIndex")
  .fit(df)
val indexed = indexer.transform(df)


val encoder = new OneHotEncoder().setInputCol("mnemonicIndex").
  setOutputCol("mnemonicVec")
val encoded = encoder.transform(indexed)
val data = encoded.select("mnemonicVec", "average", "median", "stddev")


val parsedData = data.map(row => LabeledPoint(row.getDouble(0), row.getAs[Vector](1)))

<console>:297: error: kinds of the type arguments (Vector) do not conform to the expected kinds of the type parameters (type T). Vector's type parameters do not match type T's expected parameters: type Vector has one type parameter, but type T has none val parsedData = data.map(row => LabeledPoint(row.getDouble(0), row.getAs[Vector](1))

Re: Type Error when attempting Linear Regression

dkumar1 — Fri, 11 Dec 2015 13:18:25 GMT

Which version of Spark and HDP are you using?

Re: Type Error when attempting Linear Regression

jwiden — Fri, 11 Dec 2015 14:19:11 GMT

Vedant, give this a shot:

val parsedData = data.map(row => LabeledPoint(row.getDouble(0), row.asInstanceOf[Vector](1)))

Re: Type Error when attempting Linear Regression

vjain — Sat, 12 Dec 2015 03:15:20 GMT

@Dhruv Kumar

Spark1.4.1 and HDP2.3.2

Re: Type Error when attempting Linear Regression

vjain — Sat, 12 Dec 2015 03:16:47 GMT

@Joe Widen I tried it earlier and gave me the same error.

Re: Type Error when attempting Linear Regression

dkumar1 — Sun, 13 Dec 2015 06:04:03 GMT

In addition to Vectors, you need to import the Spark Vector class explicitly since Scala imports its in-built Vector type by default. Try this:

import org.apache.spark.mllib.linalg.{Vector, Vectors}

Re: Type Error when attempting Linear Regression

vjain — Mon, 14 Dec 2015 22:20:22 GMT

@Dhruv Kumar Thanks it worked.