Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

SPARK HIVE - Parquet and Snappy format - Table issue

avatar
Contributor

I am trying to create a hive table in parquet format with snappy compression. Instead of sqlContext I am using HiveContext to directly save my dataframe results into a table using saveAsTable("<table name>").

I set the format using "hc.setConf('spark.sql.parquet.compression.codec','snappy')"

But the hive table is always created as parquet with gz compression instead of parquet with snappy compression codec. Is there any solution for this?

2 REPLIES 2

avatar
Master Guru

@Mahendiran Palani Samy

Try with .option instead of hc.setConf

Example:

dataframe.write()
.format("parquet")
.option("compression","snappy")
.saveAsTable("<table_name>")


avatar
Contributor

Thanks, Shu.. It is working now