Support Questions

Find answers, ask questions, and share your expertise
Announcements
Welcome to the upgraded Community! Read this blog to see What’s New!

ParquetFileFormat could not be instantiated while saving

avatar
New Contributor

Hi!

We have migrated from Spark 1.6 to 2.4 getting the follow error when saving a dataframe into a parquet file:

org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat could not be instantiated

 

When we executed this code:

 

StructType schema = QueryResultsHelper.createStructType(fields);
Dataset<Row> df = sqlContext.createDataFrame(rows, schema);
df.coalesce(1).write().format("parquet").mode(SaveMode.Append).save(stringPath);

 

It fails when saving. I understand is due to the migration. Could you give me some advice to solve this?

Thank you very much.

0 REPLIES 0
Labels