Created 04-18-2019 10:28 PM
I am trying to create a hive table in parquet format with snappy compression. Instead of sqlContext I am using HiveContext to directly save my dataframe results into a table using saveAsTable("<table name>").
I set the format using "hc.setConf('spark.sql.parquet.compression.codec','snappy')"
But the hive table is always created as parquet with gz compression instead of parquet with snappy compression codec. Is there any solution for this?
Created 04-19-2019 12:53 AM
Try with .option instead of hc.setConf
Example:
dataframe.write()
.format("parquet")
.option("compression","snappy")
.saveAsTable("<table_name>")
Created 04-25-2019 08:27 PM
Thanks, Shu.. It is working now