I was able to fix this in our CDP 7.1.4 cluster today by disabling the Enable Optimized S3 Committers - spark.cloudera.s3_committers.enabled in the Spark Service Configuration This works for me because we are using HDFS on premise. If you are using S3, I'm guessing that this is put in place because of the S3 eventual consistency issues. I've then also added the spark.sql.sources.partitionOverwriteMode=dynamic setting to my spark-defaults.conf also in Spark Service Configuration via the Safety Valve settings. ... View more

I'm having this same issue whether I specify this config in the spark-defaults.conf via Cloudera Manager for CDP 7.1.4 or inline in my spark.write.option("partitionOverwriteMode", "dynamic"). Error message is:  java.io.IOException: PathOutputCommitProtocol does not support dynamicPartitionOverwrite ... View more