I was getting the same error after Cloudera upgradation while using insert overwrite with config spark.sql.sources.partitionOverwriteMode=dynamic. For me below config property resolved the issue. "spark.sql.sources.commitProtocolClass=org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol" Since after upgrade default value was "spark.sql.sources.commitProtocolClass= org.apache.spark.internal.io.cloud.PathOutputCommitProtocol" and it was creating issue. ... View more