Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

toJSON is not a member of org.apache.spark.sql.SchemaRDD

toJSON is not a member of org.apache.spark.sql.SchemaRDD

Explorer

Hi,

 

I'm trying to read a hive table using HiveContext and built a jar-file using SBT tool. I am getting the below error.

value toJSON is not a member of org.apache.spark.sql.SchemaRDD

 

But, it works well in spark-shell.

 

 

Version:

  Spark 1.2.1

 

My sbt file has the below content

  

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.1"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0"
libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.1.0"

 

 

My source code is:

 


import org.apache.spark.SparkConf
import org.apache.spark.SparkContext._
import org.apache.spark.SparkContext
import org.apache.spark.sql.hive.HiveContext

object DeptJson {
def main(args:Array[String]) {
val conf = new SparkConf().setAppName("jeeJson")
val sc = new SparkContext(conf)
val hc = new HiveContext(sc)

val dept = hc.sql("select * from departments")

dept.toJSON.saveAsTextFile("dept.json")

}

}

 

I'd appreciate if someone could help on this.

 

Thanks...

 

 

3 REPLIES 3

Re: toJSON is not a member of org.apache.spark.sql.SchemaRDD

Contributor

Its because the version of Spark SQL you are using (1.0.0) has not the toJSON() method. Is there any specific reason behind using these old versions? 

 

When I do:

Umeshs-MacBook-Pro:~ umesh$ javap -classpath .m2/repository/org/apache/spark/spark-sql_2.10/1.2.1/spark-sql_2.10-1.2.1.jar org/apache/spark/sql/SchemaRDD | grep 'toJSON'
  public org.apache.spark.rdd.RDD<java.lang.String> toJSON();
Umeshs-MacBook-Pro:~ umesh$ 

I can see toJSON method in SparkSQL v 1.2.1 

 

but when I do:

Umeshs-MacBook-Pro:~ umesh$ javap -classpath Downloads/spark-sql_2.10-1.0.0.jar org/apache/spark/sql/SchemaRDD | grep 'toJSON'
Umeshs-MacBook-Pro:~ umesh$ 

I cant see the method.

 

So start using SparkSQL v 1.2.1 at least for getting this method and it will comply you Spark Core version also.

Re: toJSON is not a member of org.apache.spark.sql.SchemaRDD

Explorer

Hi Umesh,

 Thank you so much for your reply and  your effort to take to read my question.. I plan to take CCA175 certification and i'm practising in the cloudera Quicstart VM cluster with Spark-1.2.1. The same script works fine in spark-shell, But it doesn't work in the application jar file. Could i use spark-sql 1.2.1 with spark-1.2.1? I'm awaiting for your response.

 

 

Thanks,

 

Highlighted

Re: toJSON is not a member of org.apache.spark.sql.SchemaRDD

Contributor

Yes you can!

Don't have an account?
Coming from Hortonworks? Activate your account here