Product Announcements
Find the latest product announcements and version updates
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[ANNOUNCE] CDS 3.0 Powered by Apache Spark 3 is now GA!

Highlighted

[ANNOUNCE] CDS 3.0 Powered by Apache Spark 3 is now GA!

Cloudera Employee

We are happy to announce General Availability of CDS 3.0 Powered By Apache Spark 3.0.1.

 

You can download the parcel and apply it directly to provisioned clusters without disrupting your currently running Spark workloads, while taking advantage of all new features and benefits that come with Spark 3.0.

 

This component is generally available and is supported on CDP Private Cloud Base clusters running version 7.1.3 and above.

 

What's New in CDS 3.0?

 

  • Support of JDK11, Scala 2.12, Python 3.4+ (Python 2.7+ deprecated)
  • Adaptive execution of Spark SQL
  • Dynamic Partition Pruning
  • Binary files data source
  • DataSource V2 Improvements (Pluggable catalog integration)
  • Structured Streaming UI
  • Auto discover and schedule tasks on nodes with GPUs on a YARN cluster
  • Kafka connector delegation token (0.10+)

 

See documentation for details and installation.

 

--------------

Want to become a pro Spark user?  Sign up for Apache Spark Training.

Don't have an account?
Coming from Hortonworks? Activate your account here