Support Questions
Find answers, ask questions, and share your expertise

ParquetFileFormat could not be instantiated while saving

New Contributor

Hi!

We have migrated from Spark 1.6 to 2.4 getting the follow error when saving a dataframe into a parquet file:

org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat could not be instantiated

 

When we executed this code:

 

StructType schema = QueryResultsHelper.createStructType(fields);
Dataset<Row> df = sqlContext.createDataFrame(rows, schema);
df.coalesce(1).write().format("parquet").mode(SaveMode.Append).save(stringPath);

 

It fails when saving. I understand is due to the migration. Could you give me some advice to solve this?

Thank you very much.

0 REPLIES 0
; ;