Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to insert data into Hive from SparkSQL

Highlighted

How to insert data into Hive from SparkSQL

New Contributor

Below is my code

import sqlContext.implicits._

import org.apache.spark.sql

val eBayText = sc.textFile("/user/cloudera/spark/servicesDemo.csv")

val hospitalDataText = sc.textFile("/user/cloudera/spark/servicesDemo.csv")

val header = hospitalDataText.first()

val hospitalData = hospitalDataText.filter(a=>a!=header)

case class Services(uhid:String,locationid:String,doctorid:String)

val hData = hospitalData.map(_.split(",")).map(p=>Services(p(0),p(1),p(2)))

val hosService = hData.toDF()

hosService.write.format("parquet").mode(org.apache.spark.sql.SaveMode.Append).save("/user/hive/warehouse/hosdata")

This code created 'hosdata' folder at specified path, which contains data in 'parquet' format.

But when i went to hive and check table got created or not the, i did not able to see any table name as 'hosdata'.

So i run below commands.

hosService.write.mode("overwrite").saveAsTable("hosData")

sqlContext.sql("show tables").show

shows me below result

+--------------------+-----------+

| tableName|isTemporary|

+--------------------+-----------+

| hosdata| false|

+--------------------+-----------+

But again when i check in hive, i can not see table 'hosdata'

Could anyone let me know what step i am missing?