We are happy to announce General Availability of CDS 3.0 Powered By Apache Spark 3.0.1.
You can download the parcel and apply it directly to provisioned clusters without disrupting your currently running Spark workloads, while taking advantage of all new features and benefits that come with Spark 3.0.
This component is generally available and is supported on CDP Private Cloud Base clusters running version 7.1.3 and above.
What's New in CDS 3.0?
- Support of JDK11, Scala 2.12, Python 3.4+ (Python 2.7+ deprecated)
- Adaptive execution of Spark SQL
- Dynamic Partition Pruning
- Binary files data source
- DataSource V2 Improvements (Pluggable catalog integration)
- Structured Streaming UI
- Auto discover and schedule tasks on nodes with GPUs on a YARN cluster
- Kafka connector delegation token (0.10+)
See documentation for details and installation.
--------------
Want to become a pro Spark user? Sign up for Apache Spark Training.
