Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎08-02-2018

Unable to write the data into hive using spark-submit

Hi,

 

I developed a simple spark application using scala which streams the data from Kafka topic and writes it into the hive when I run this application in the spark shell it is working fine, but when I try to submit the same application using the spark submit I am not able to write the data into hive.

 

 

Spark-Submit

 

./spark-submit --class cloudera1.cloudera1 --master local[2] /root/testjars/XXXXXXcloudera1/target/XXXXX.cloudera1-1.0-SNAPSHOT.jar XXXXX_XXXXXX default. testtable_31 false

 

 

Below is the code which writes the data into Hive,

 

lines.foreachRDD{rdd=>

if (rdd.isEmpty) {println(" ")}

      else{

        val sqlContext = new org.apache.spark.sql.SQLContext(sc)

         import spark.implicits._

        val dataframe = spark.read.json(parseAvroRdd(rdd))

        val dataframe1= dataframe.toDF()

        dataframe.show()

        dataframe.write.mode(SaveMode.Append).saveAsTable("testtable_31")

      }

}

    ssc.start()

    ssc.awaitTermination()

  }

}

 

Thanks in advance.