Support Questions
Find answers, ask questions, and share your expertise

CDS 2.4 Spark partionBy clause doesn't work with OSS delta lake tables

New Contributor

The fix for Spark-27453 is yet to be included in the CDS 2.4x release which resolves the partition drop issue for Spark dataframe partitionBy clause for external sources such as delta lake tables. For Spark versions 2.4.3 and above this patch is included. Any plans to include the same in the CDS release? Without partitioning, delta lake use cases are difficult to materialise on cloudera distributions