Hi, I'm trying to read a hive table using HiveContext and built a jar-file using SBT tool. I am getting the below error. value toJSON is not a member of org.apache.spark.sql.SchemaRDD But, it works well in spark-shell. Version: Spark 1.2.1 My sbt file has the below content name := "Simple Project" version := "1.0" scalaVersion := "2.10.4" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.1" libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.0.0" libraryDependencies += "org.apache.spark" % "spark-hive_2.10" % "1.1.0" My source code is: import org.apache.spark.SparkConf import org.apache.spark.SparkContext._ import org.apache.spark.SparkContext import org.apache.spark.sql.hive.HiveContext object DeptJson { def main(args:Array[String]) { val conf = new SparkConf().setAppName("jeeJson") val sc = new SparkContext(conf) val hc = new HiveContext(sc) val dept = hc.sql("select * from departments") dept.toJSON.saveAsTextFile("dept.json") } } I'd appreciate if someone could help on this. Thanks... ... View more

Hi, what does the code do RDD.map(e => e.split(",")(1)) in spark with scala? I know RDD.map(e => e.split(",")) would split a string with ',' separated. But, I don't get the (1) in the code. Could somebody give me some information on this? I would greatly appreciate your answer. Thanks! ... View more