Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

toJSON is not a member of org.apache.spark.sql.SchemaRDD

avatar
Rising Star

Hi,

I'm trying to read a hive table using HiveContext and built a jar-file using SBT tool. I am getting the below error.

value toJSON is not a member of org.apache.spark.sql.SchemaRDD

But, it works well in spark-shell.

Version:

Spark 1.2.1

My sbt file has the below content

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.1"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.1.0"

My source code is:

import org.apache.spark.SparkConf import org.apache.spark.SparkContext._ import org.apache.spark.SparkContext import org.apache.spark.sql.hive.HiveContext

object DeptJson { def main(args:Array[String]) { val conf = new SparkConf().setAppName("jeeJson") val sc = new SparkContext(conf) val hc = new HiveContext(sc)

val dept = hc.sql("select * from departments")

dept.toJSON.saveAsTextFile("dept.json")

}

}

I'd appreciate if someone could help on this.

Thanks...

1 ACCEPTED SOLUTION

avatar
Guru

@Jeeva Jeeva

Your library dependency for SparkSQL looks to be 1.0.0.

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.1.0"

org.apache.spark.rdd.RDD did not acquire the toJson method until 1.2. Load Spark-SQL 1.2 and then give it another try.

View solution in original post

4 REPLIES 4

avatar
Expert Contributor

In the latest version of Hortonworks sandbox 2.4 (you can download for free from hortonworks.com) zeppelin and spark run out of the box. Spark version is 1.6 and the toJSON method works, make sure you run it on a DataFrame not RDD

val js = tradesRDD.toDF.toJSON
js.take(2)
output:
Array[String] = 
Array({"trader":"Kirk","price":11.0,"qty":51,"vol":40000,"product":"goog","time":"2016-03-29
 10:38:12.0"}, 
{"trader":"Kirk","price":0.0,"qty":66,"vol":40000,"product":"goog","time":"2016-03-29
 10:56:12.0"})

avatar
Rising Star

Hi Kirk Haslbeck,

Thanks for your reply. But, I have to fulfill this through Spark 1.2.1. I greatly appreciate your reply..

avatar
Guru

@Jeeva Jeeva

Your library dependency for SparkSQL looks to be 1.0.0.

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.1.0"

org.apache.spark.rdd.RDD did not acquire the toJson method until 1.2. Load Spark-SQL 1.2 and then give it another try.

avatar
Rising Star

Thanks for your reply and i will try this and let you know if it works. I greatly appreciate your your effort and time.....