[ANNOUNCE] CDS 3.1 Powered by Apache Spark 3.1.1 is now GA!

[#1]

CDS 3.1 powered by Apache Spark 3.1.1 is now generally available for CDP Private Cloud Base 7.1.6.

This is a minor release of CDS 3. The main improvements include:

• The parcel contains spark3 compliant spark-hbase connector

• All performance enhancement of Apache Spark 3.1.1 such as new optimizer rules and improved subexpression eliminations

• Unify create table SQL syntax

• Shuffled hash join improvement

It can be installed as an add-on parcel on top of CDP Private Cloud Base 7.1.6. The bits can be found here and the latest documentation can be found here. 

For Public Cloud the same bits will be available in a pre-warmed image for Cloudera Runtime 7.2.9. That can be installed through Data Engineering Spark3 Data Hub cluster templates.

--------------

Want to become a pro Spark user? Sign up for Apache Spark Training.