Insert Spark dataframe into hive partitioned

[#1]

@hem lohani

Try with below syntax

df.write.mode("<append or overwrite>").partitionBy("<partition_cols>").insertInto("<hive_table_name>")

[#2]

@hem lohani

Create column using withColumn function with literal value as 12.

Use month column as partitionby column and use insertInto table.

df.withColumn("month",lit(12)).write.mode("<append or overwrite>").partitionBy("month").insertInto("<hive_table_name>") 

(or)

Using SQL query

df.createOrReplaceTempView("temp_table")
spark.sql("insert into <partition_table> partition(`month`=12) select * from <temp_table>")

-

If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂

[#3]

@hem lohani

--> Permission Denied - Please check the permission on the HDFS directory!

--> Could you share the error logs, of the error that you are getting?

[#4]

I need to insert for a specific parttion lets say month =12 , how will i mention partition value?

[#5]

Thnaks, but i am already doing this in HDP2 and it worked, in HDP3.0 and hive 3.0 it is giving error when i am writing spark dataframe into hive portioned table . it says permission denied. any help?

[#6]

you can’t execute an “INSERT OVERWRITE PARTITION, in hew HDP3.0

[#7]

i have permissions but stil getting error