Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Insert Spark dataframe into hive partitioned

avatar
Explorer
 
7 REPLIES 7

avatar
Master Guru

@hem lohani

Try with below syntax

df.write.mode("<append or overwrite>").partitionBy("<partition_cols>").insertInto("<hive_table_name>")


avatar
Master Guru

@hem lohani

Create column using withColumn function with literal value as 12.

Use month column as partitionby column and use insertInto table.

df.withColumn("month",lit(12)).write.mode("<append or overwrite>").partitionBy("month").insertInto("<hive_table_name>") 

(or)

Using SQL query

df.createOrReplaceTempView("temp_table")
spark.sql("insert into <partition_table> partition(`month`=12) select * from <temp_table>")

-

If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂

avatar
Master Guru

@hem lohani

--> Permission Denied - Please check the permission on the HDFS directory!

--> Could you share the error logs, of the error that you are getting?

avatar
Explorer

I need to insert for a specific parttion lets say month =12 , how will i mention partition value?

avatar
Explorer

Thnaks, but i am already doing this in HDP2 and it worked, in HDP3.0 and hive 3.0 it is giving error when i am writing spark dataframe into hive portioned table . it says permission denied. any help?

avatar
Explorer

you can’t execute an “INSERT OVERWRITE PARTITION, in hew HDP3.0

avatar
Explorer

i have permissions but stil getting error