- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
SaveAsTable not creating table in hive metastore
- Labels:
-
Apache Hadoop
-
Apache Hive
-
Apache Spark
Created ‎05-14-2019 02:50 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am trying to save json data in hive table as follows:
input.write.mode(SaveMode.Overwrite).format("json").partitionBy("col1", "col2").saveAsTable("data")
it is creating data under spark warehouse dir in HDFS but no table gets created in hive metastore. I am using spark 2.3.0. Can someone help with this?
Created ‎05-15-2019 01:35 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Could you once make sure you have enabled hive support while creating spark session.
Ex: enabling hive support
val spark = SparkSession
.builder()
.appName("app_name")
.config("spark.sql.warehouse.dir", warehouseLocation)
.enableHiveSupport()
.getOrCreate()
Refer to this link for more details regards enableHiveSupport.
Created ‎05-15-2019 04:54 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Shu, Thanks for responding. I tried to update ticket earlier but, couldn't happen. I am not sure if some configs which I had set were causing this issue but, after commenting those out it started working fine. Below are the configs:
.config("spark.memory.fraction", "0.3") .config("spark.storage.memoryFraction", "0.8") .config("spark.executor.heartbeatInterval", "300") .config("spark.network.timeout", "900") .config("spark.sql.autoBroadcastJoinThreshold", "-1")
