question toJSON is not a member of org.apache.spark.sql.SchemaRDD in Archives of Support Questions (Read Only)

toJSON is not a member of org.apache.spark.sql.SchemaRDD

JeevaKrish — Wed, 27 Apr 2016 23:57:17 GMT

Hi,

I'm trying to read a hive table using HiveContext and built a jar-file using SBT tool. I am getting the below error.

value toJSON is not a member of org.apache.spark.sql.SchemaRDD

But, it works well in spark-shell.

Version:

Spark 1.2.1

My sbt file has the below content

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.1"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.1.0"

My source code is:

import org.apache.spark.SparkConf import org.apache.spark.SparkContext._ import org.apache.spark.SparkContext import org.apache.spark.sql.hive.HiveContext

object DeptJson { def main(args:Array[String]) { val conf = new SparkConf().setAppName("jeeJson") val sc = new SparkContext(conf) val hc = new HiveContext(sc)

val dept = hc.sql("select * from departments")

dept.toJSON.saveAsTextFile("dept.json")

}

I'd appreciate if someone could help on this.

Thanks...

Re: toJSON is not a member of org.apache.spark.sql.SchemaRDD

khaslbeck — Thu, 28 Apr 2016 01:18:57 GMT

In the latest version of Hortonworks sandbox 2.4 (you can download for free from hortonworks.com) zeppelin and spark run out of the box. Spark version is 1.6 and the toJSON method works, make sure you run it on a DataFrame not RDD

val js = tradesRDD.toDF.toJSON
js.take(2)
output:
Array[String] = 
Array({"trader":"Kirk","price":11.0,"qty":51,"vol":40000,"product":"goog","time":"2016-03-29
 10:38:12.0"}, 
{"trader":"Kirk","price":0.0,"qty":66,"vol":40000,"product":"goog","time":"2016-03-29
 10:56:12.0"})

Re: toJSON is not a member of org.apache.spark.sql.SchemaRDD

JeevaKrish — Thu, 28 Apr 2016 07:46:15 GMT

Hi Kirk Haslbeck,

Thanks for your reply. But, I have to fulfill this through Spark 1.2.1. I greatly appreciate your reply..

Re: toJSON is not a member of org.apache.spark.sql.SchemaRDD

vvaks — Thu, 28 Apr 2016 19:43:15 GMT

@Jeeva Jeeva

Your library dependency for SparkSQL looks to be 1.0.0.

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.1.0"

org.apache.spark.rdd.RDD did not acquire the toJson method until 1.2. Load Spark-SQL 1.2 and then give it another try.

Re: toJSON is not a member of org.apache.spark.sql.SchemaRDD

JeevaKrish — Sat, 30 Apr 2016 04:08:08 GMT

Thanks for your reply and i will try this and let you know if it works. I greatly appreciate your your effort and time.....