ParquetFileFormat could not be instantiated while saving

[#1]

Hi!

We have migrated from Spark 1.6 to 2.4 getting the follow error when saving a dataframe into a parquet file:

org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat could not be instantiated

When we executed this code:

StructType schema = QueryResultsHelper.createStructType(fields);
Dataset<Row> df = sqlContext.createDataFrame(rows, schema);
df.coalesce(1).write().format("parquet").mode(SaveMode.Append).save(stringPath);

It fails when saving. I understand is due to the migration. Could you give me some advice to solve this?

Thank you very much.

[#2]

HI @kerbero can you provide us the full error trace.

It could be due to some dependency issue when using spark2.x