Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

SaveAsTable not creating table in hive metastore

New Contributor

I am trying to save json data in hive table as follows:

input.write.mode(SaveMode.Overwrite).format("json").partitionBy("col1", "col2").saveAsTable("data")

it is creating data under spark warehouse dir in HDFS but no table gets created in hive metastore. I am using spark 2.3.0. Can someone help with this?

2 REPLIES 2

Super Guru

@Aman Rastogi

Could you once make sure you have enabled hive support while creating spark session.

Ex: enabling hive support

val spark = SparkSession
  .builder()
  .appName("app_name")
  .config("spark.sql.warehouse.dir", warehouseLocation)
  .enableHiveSupport()
  .getOrCreate()

Refer to this link for more details regards enableHiveSupport.

New Contributor

@Shu, Thanks for responding. I tried to update ticket earlier but, couldn't happen. I am not sure if some configs which I had set were causing this issue but, after commenting those out it started working fine. Below are the configs:


.config("spark.memory.fraction", "0.3")
.config("spark.storage.memoryFraction", "0.8")
.config("spark.executor.heartbeatInterval", "300")
.config("spark.network.timeout", "900")
.config("spark.sql.autoBroadcastJoinThreshold", "-1")
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.