Hi! We have migrated from Spark 1.6 to 2.4 getting the follow error when saving a dataframe into a parquet file: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat could not be instantiated When we executed this code: StructType schema = QueryResultsHelper . createStructType ( fields ) ; Dataset < Row > df = sqlContext .createDataFrame( rows , schema ) ; df .coalesce( 1 ).write().format( "parquet" ).mode( SaveMode.Append ).save( stringPath ) ; It fails when saving. I understand is due to the migration. Could you give me some advice to solve this? Thank you very much.
... View more
