Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

SaveAsTable not creating table in hive metastore

avatar

I am trying to save json data in hive table as follows:

input.write.mode(SaveMode.Overwrite).format("json").partitionBy("col1", "col2").saveAsTable("data")

it is creating data under spark warehouse dir in HDFS but no table gets created in hive metastore. I am using spark 2.3.0. Can someone help with this?

2 REPLIES 2

avatar
Master Guru

@Aman Rastogi

Could you once make sure you have enabled hive support while creating spark session.

Ex: enabling hive support

val spark = SparkSession
  .builder()
  .appName("app_name")
  .config("spark.sql.warehouse.dir", warehouseLocation)
  .enableHiveSupport()
  .getOrCreate()

Refer to this link for more details regards enableHiveSupport.

avatar

@Shu, Thanks for responding. I tried to update ticket earlier but, couldn't happen. I am not sure if some configs which I had set were causing this issue but, after commenting those out it started working fine. Below are the configs:


.config("spark.memory.fraction", "0.3")
.config("spark.storage.memoryFraction", "0.8")
.config("spark.executor.heartbeatInterval", "300")
.config("spark.network.timeout", "900")
.config("spark.sql.autoBroadcastJoinThreshold", "-1")