Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SaveAsTable not creating table in hive metastore

SaveAsTable not creating table in hive metastore

New Contributor

I am trying to save json data in hive table as follows:

input.write.mode(SaveMode.Overwrite).format("json").partitionBy("col1", "col2").saveAsTable("data")

it is creating data under spark warehouse dir in HDFS but no table gets created in hive metastore. I am using spark 2.3.0. Can someone help with this?

2 REPLIES 2
Highlighted

Re: SaveAsTable not creating table in hive metastore

Super Guru

@Aman Rastogi

Could you once make sure you have enabled hive support while creating spark session.

Ex: enabling hive support

val spark = SparkSession
  .builder()
  .appName("app_name")
  .config("spark.sql.warehouse.dir", warehouseLocation)
  .enableHiveSupport()
  .getOrCreate()

Refer to this link for more details regards enableHiveSupport.

Highlighted

Re: SaveAsTable not creating table in hive metastore

New Contributor

@Shu, Thanks for responding. I tried to update ticket earlier but, couldn't happen. I am not sure if some configs which I had set were causing this issue but, after commenting those out it started working fine. Below are the configs:


.config("spark.memory.fraction", "0.3")
.config("spark.storage.memoryFraction", "0.8")
.config("spark.executor.heartbeatInterval", "300")
.config("spark.network.timeout", "900")
.config("spark.sql.autoBroadcastJoinThreshold", "-1")
Don't have an account?
Coming from Hortonworks? Activate your account here