I am trying to load a CSV or an XML File using Intellij Spark Scala into a pre-existing hive table and then it gives below exceptions on the last step while saving dataframe.

Ironically: the code below works fine in spark-shell without any issues with all four cases. 1. when I use Hive Context and Insertinto().

val sparkConf = new SparkConf().setAppName("TEST")
val sc = new SparkContext(sparkConf)
val hiveContext = new HiveContext(sc)hiveContext.setConf("hive.exec.dynamic.partition", "true")hiveContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict")println("CONFIG DONE!!!!!")
val xml = hiveContext.read.format("com.databricks.spark.xml").option("rowTag","employee").load("/PUBLIC_TABLESPACE/updatedtest1.xml")println("XML LOADED!!!!!!")xml.write.format("parquet").mode("overwrite").partitionBy("designation").insertInto("test2")println("TABLE SAVED!!!!!!!")

Exception in thread "main" java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(org.apache.hadoop.fs.Path, java.lang.String, java.util.Map, boolean, int, boolean, boolean, boolean)

2.when I use Hive Context and SaveAsTable().

val sparkConf = new SparkConf().setAppName("TEST")
val sc = new SparkContext(sparkConf)
val hiveContext = new HiveContext(sc)hiveContext.setConf("hive.exec.dynamic.partition", "true")hiveContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict")println("CONFIG DONE!!!!!")
val xml = hiveContext.read.format("com.databricks.spark.xml").option("rowTag","employee").load("/PUBLIC_TABLESPACE/updatedtest1.xml")println("XML LOADED!!!!!!")

xml.write.format("parquet").mode("overwrite").partitionBy("designation").saveAsTable("test2")

Exception in thread "main" java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(org.apache.hadoop.fs.Path, java.lang.String, java.util.Map, boolean, int, boolean, boolean, boolean)

3. when I use SQL Context and Insertinto().

val sparkConf = new SparkConf().setAppName("TEST")
val sc = new SparkContext(sparkConf)
val hiveContext = new SQLContext(sc)hiveContext.setConf("hive.exec.dynamic.partition", "true")hiveContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict")println("CONFIG DONE!!!!!")
val xml = hiveContext.read.format("com.databricks.spark.xml").option("rowTag","employee").load("/PUBLIC_TABLESPACE/updatedtest1.xml")println("XML LOADED!!!!!!") xml.write.format("parquet").mode("overwrite").partitionBy("designation").insertInto("test2")println("TABLE SAVED!!!!!!!")

Exception in thread "main" org.apache.spark.sql.AnalysisException: Table not found: test2;

4. when I use SQL Context and saveAsTable().

val sparkConf = new SparkConf().setAppName("TEST")
val sc = new SparkContext(sparkConf)
val hiveContext = new SQLContext(sc)hiveContext.setConf("hive.exec.dynamic.partition", "true")hiveContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict")println("CONFIG DONE!!!!!") 
val xml = hiveContext.read.format("com.databricks.spark.xml").option("rowTag","employee").load("/PUBLIC_TABLESPACE/updatedtest1.xml")println("XML LOADED!!!!!!") xml.write.format("parquet").mode("overwrite").partitionBy("designation").saveAsTable("test2")println("TABLE SAVED!!!!!!!")

Exception in thread "main" java.lang.RuntimeException: Tables created with SQLContext must be TEMPORARY. Use a HiveContext instead.