toJSON is not a member of org.apache.spark.sql.SchemaRDD

[#1]

Hi,

I'm trying to read a hive table using HiveContext and built a jar-file using SBT tool. I am getting the below error.

value toJSON is not a member of org.apache.spark.sql.SchemaRDD

But, it works well in spark-shell.

Version:

Spark 1.2.1

My sbt file has the below content

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.1"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.1.0"

My source code is:

import org.apache.spark.SparkConf import org.apache.spark.SparkContext._ import org.apache.spark.SparkContext import org.apache.spark.sql.hive.HiveContext

object DeptJson { def main(args:Array[String]) { val conf = new SparkConf().setAppName("jeeJson") val sc = new SparkContext(conf) val hc = new HiveContext(sc)

val dept = hc.sql("select * from departments")

dept.toJSON.saveAsTextFile("dept.json")

}

}

I'd appreciate if someone could help on this.

Thanks...

[#2]

In the latest version of Hortonworks sandbox 2.4 (you can download for free from hortonworks.com) zeppelin and spark run out of the box. Spark version is 1.6 and the toJSON method works, make sure you run it on a DataFrame not RDD

val js = tradesRDD.toDF.toJSON
js.take(2)
output:
Array[String] = 
Array({"trader":"Kirk","price":11.0,"qty":51,"vol":40000,"product":"goog","time":"2016-03-29
 10:38:12.0"}, 
{"trader":"Kirk","price":0.0,"qty":66,"vol":40000,"product":"goog","time":"2016-03-29
 10:56:12.0"})

[#3]

Hi Kirk Haslbeck,

Thanks for your reply. But, I have to fulfill this through Spark 1.2.1. I greatly appreciate your reply..

[#4 · Accepted solution]

@Jeeva Jeeva

Your library dependency for SparkSQL looks to be 1.0.0.

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.1.0"

org.apache.spark.rdd.RDD did not acquire the toJson method until 1.2. Load Spark-SQL 1.2 and then give it another try.

[#5]

Thanks for your reply and i will try this and let you know if it works. I greatly appreciate your your effort and time.....