- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Insert Spark dataframe into hive partitioned
Created ‎06-29-2019 02:16 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎06-30-2019 02:58 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Try with below syntax
df.write.mode("<append or overwrite>").partitionBy("<partition_cols>").insertInto("<hive_table_name>")
Created ‎06-30-2019 06:03 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Create column using withColumn function with literal value as 12.
Use month column as partitionby column and use insertInto table.
df.withColumn("month",lit(12)).write.mode("<append or overwrite>").partitionBy("month").insertInto("<hive_table_name>")
(or)
Using SQL query
df.createOrReplaceTempView("temp_table") spark.sql("insert into <partition_table> partition(`month`=12) select * from <temp_table>")
-
If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂
Created ‎07-01-2019 03:50 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
--> Permission Denied - Please check the permission on the HDFS directory!
--> Could you share the error logs, of the error that you are getting?
Created ‎06-30-2019 09:28 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I need to insert for a specific parttion lets say month =12 , how will i mention partition value?
Created ‎06-30-2019 06:31 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thnaks, but i am already doing this in HDP2 and it worked, in HDP3.0 and hive 3.0 it is giving error when i am writing spark dataframe into hive portioned table . it says permission denied. any help?
Created ‎07-02-2019 07:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
you can’t execute an “INSERT OVERWRITE PARTITION, in hew HDP3.0
Created ‎07-02-2019 07:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i have permissions but stil getting error
