Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

ParquetFileFormat could not be instantiated while saving

avatar
New Contributor

Hi!

We have migrated from Spark 1.6 to 2.4 getting the follow error when saving a dataframe into a parquet file:

org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat could not be instantiated

 

When we executed this code:

 

StructType schema = QueryResultsHelper.createStructType(fields);
Dataset<Row> df = sqlContext.createDataFrame(rows, schema);
df.coalesce(1).write().format("parquet").mode(SaveMode.Append).save(stringPath);

 

It fails when saving. I understand is due to the migration. Could you give me some advice to solve this?

Thank you very much.

1 REPLY 1

avatar
Expert Contributor

HI @kerbero can you provide us the full error trace.

It could be due to some dependency issue when using spark2.x