What's New @ Cloudera

As part of the changes announced in October 2020, beginning in February 2021, no versions of Cloudera Manager, CDH, Apache Ambari, HDP and HDF releases will be available via Cloudera's public repositories.  Instead, all of the then-currently supported releases of Cloudera Manager, CDH, Ambari, HDP and HDF will be accessible only to active customers with valid subscription entitlements to these Cloudera products. This follows similar changes made for versions released since November 2019, which were never posted in Cloudera’s public repositories.     Additionally, new patch releases of Cloudera Manager and Ambari have been made available and are required to access Cloudera’s private repositories, which now contain these new and existing releases. Versions other than those outlined below will no longer be able to install CDH, HDP or HDF from Cloudera’s public repositories.  Customers should strongly consider upgrading to one of the new Cloudera Manager or Ambari

Text classification is a ubiquitous capability with a wealth of use cases including sentiment analysis, topic assignment, document identification, article recommendation, and more. But collecting enough annotated examples to train traditional classifiers can be quite costly. Instead, we take a look at a classic technique that can be used to perform text classification with few or even zero training examples! We're talking about text embeddings, of course. New advances have significantly increased the quality of document embeddings and in our newest writing on Few Shot Text Classification  we coverhow to use them for topic classification,best practices for using them,and potential limitations.Follow the links in the report to find code snippets so you can try it for yourself, and build your own demo so you can see the method in action!

 We are pleased to announce the release of Cloudera JDBC 2.6.13 driver for Apache Hive.The release has the following fixes and enhancements: Enhancements and New Features[HIVEJ-566] The driver now supports Apache Hive versions 1.0.0 through 3.1, and CDH versions 5.10 through 5.16 and 6.0 through 6.3. [HIVEJ-604] The driver now supports Java 11.  Resolved Issues[00184182][HIVEJ-602] The driver logs incorrect socket timeout values. [00255458][HIVEJ-637] When the connection is not valid, the driver does not notify the ConnectionEventListener object as expected. Known Issues The following are known issues that you may encounter due to limitations in the data source, the driver, or an application. [HIVEJ-577] The JDBC 4.0 version of the driver fails to connect to servers that require encryption using TLS 1.1 or later. When you attempt to connect to the server, the connection fails and the driver returns an SSL handshake exception. This issue occu

 Cloudera Fast Forward are happy to share recent updates of two of our older reports, now freely available to all!Semantic Image Search We recently revisited the topic of semantic search on image data. We’ve previously studied applications of deep learning to images in our early report Deep Learning for Image Analysis, and our more recent update: Deep Learning for Image Analysis: 2019 edition.In our most recent research cycle, we explored two critical requirements of semantic search at scale. First, we wrote a review of strategies for creating semantic representations of images (including supervised, self supervised, and unsupervised methods). Second, we provide an implementation of semantic search using fast approximate nearest neighbor search (with FAISS).We have released an updated version of ConvNet Playground App, and a set of scripts and tutorials for implementing semantic image search on the Cloudera Machine Learning platform.Federated Learning  Two years ago

New, lightweight and customizable Cloudera ML Runtimes are now generally available in CDP Machine Learning and Cloudera Data Science Workbench 1.9. ML Runtimes have been rebuilt from the ground up to enable greater flexibility in frameworks, processing, and IDEs — powering customizable lightweight deployments without over encumbering the runtime profile. The new profiles are designed to meet the diverse needs of Data Scientists by enabling a variety of ML Runtimes natively and eliminating sizing and versatility issues with previous Cloudera Engine profiles. This initial release ships with Python 3.6, 3.7, and 3.8 Runtimes as well as both the Workbench editor and with JupyterLab (Tech Preview). New Runtime options will soon follow! To learn more about our vision and roadmap read the Announcement blog post. Also in these releases: CDP Machine Learning:Support scaling down to zero CPUs or GPUs on AzureRefreshed Data Science Project dashboard experience CDSW 1.9:Av

We're pleased to announce the release of Cloudera Private Cloud (PvC) Base 7.1.5, which provides additional platform support and introduces key new features.  This is a cumulative maintenance release that carries forward the improvements from CDP PvC Base 7.1.4 and prior releases.   In this release, we’ve added the following new features and improvements: All core components of the platform are now FIPS Compliant.Improved Impala query performance using Bloom filters with a column predicate pushdown, allowing for optimized execution of filters. Clusters can now support ‘IGNORE’ operations in Apache Kudu.YARN Auto Queue Creation is now supported when Hive impersonation is turned off.Ranger support for enforcing Hive policies with HDFS access to Hive tables. Replication of Atlas metadata and Ranger policy for Hive databases (as a Tech Preview).Atlas audit now supports extended operations, such as type definitions, import, export, server start, and more.

Announcing Cloudera Enterprise 6.3.4 We are pleased to announce the release of Cloudera Enterprise 6.3.4, the world’s leading data platform for machine learning and analytics, optimized for the cloud. This maintenance release includes important bug fixes and CVE fixes since CDH-6.3.3.  Full release documentation is available here including the Cloudera Manager Release Notes and Cloudera Runtime Release Notes. New Features & CapabilitiesInstallation or Upgrade of Cloudera Manager and CDH requires authentication to access downloadsCloudera Express has been discontinuedIssues fixedUnsupported featuresIncompatible changesTimezone Names Unsupported in Impala in CDH 6.3.4Known issues and limitationsWhat's New in Cloudera Manager 6.3.4Fixed issues in Cloudera Manager 6.3.4Known Issues in Cloudera Manager 6.3.4 Important Note About Installation Beginning with Cloudera Manager 6.3.3 and CDH 6.3.3, downloading the software will require a valid Cloudera Ente

Cloudera fast forward are pleased to share our two latest applied machine learning research reports, on Meta-Learning and Structural Time Series. Read on here for more on each report,  check-out our blog post to read about our evolving research process, or head on over to our Research Roundup to view our latest research webinar on demand!Meta-Learning   In contrast to how humans learn, deep learning algorithms need vast amounts of data and compute and may yet struggle to generalize. Humans are successful in adapting quickly because they leverage their knowledge acquired from prior experience when faced with new problems. In this webinar we will explain how meta-learning can leverage previous knowledge acquired from data to solve novel tasks quickly and more efficiently during test time.Our report, Meta-Learning is freely available online, and accompanied by code that applies the technique to an image dataset. Read the report t

Model Monitoring & Deployment Security is now GA in CDSW 1.8. Available in this release is native MLOps functionalities for model monitoring, enabling tracking of individual model predictions down to the feature level for calculating model drift and ground truthing to production environments. Data scientists can analyze metrics using their preferred libraries and IDEs in any language, ensuring models are performing optimally and compliantly at scale. Additionally, this release includes general availability of Resource Quotas and quota management. These features enable administrators to limit users’ aggregate CPU, memory, and GPU consumption to protect against over-usage resulting in critical compute resource shortages. Default quotas can be configured for a Workspace, and overridden on a per-user basis with Custom quotas. Also in this release:Ability to use custom command-line arguments for jobs.Improved security for model deployments allowing user-level access controls t

We're pleased to announce the release of Cloudera Private Cloud (PvC) Base 7.1.4, which provides additional platform support and introduces key new features.  This is a cumulative maintenance release that carries forward the improvements from CDP PvC Base 7.1.3 and prior releases.   In this release, we’ve added the following new features and improvements: To help CDH and HDP customers upgrade to PvC Base, we’ve added platform support for Oracle 12, Oracle 19, SLES 12 SP5 and Postgres 11.New features to support your data warehouse with additional keywords, and replication between CDP PvC Base clusters.New features in Yarn Queue Manager to provide greater control with resource management.New security and governance features in SDX that provide multi-tenancy and translation of access control policies. See the 7.1.4 Release Summary for more details of these features.  CDH and HDP customers are encouraged to upgrade to CDP PvC Base for improved enterp

We are pleased to announce the release of Cloudera JDBC 2.6.11 driver for Apache Hive  The release has the following fixes and enhancements: Enhancements and New FeaturesThe driver now supports CDP 7.0 Resolved Issues [00222028][HIVEJ-610] Queries with a UNION clause are replaced with a UNION ALL clause. Known Issues The following are known issues that you may encounter due to limitations in the data source, the driver, or an application.  [HIVEJ-577] The JDBC 4.0 version of the driver fails to connect to servers that require encryption using TLS 1.1 or later.  When you attempt to connect to the server, the connection fails and the driver returns an SSL handshake exception. This issue occurs only when you run the driver using Java Runtime Environment (JRE) 6.0.  As a workaround, run the driver using JRE 7.0 or 8.0. Workflow ChangesSee the Release Notes for the full list of upcoming workflow changes that may disrupt the established

Summary:With the recent launch of CDP Private Cloud for on-premises deployments, we have renamed CDP Data Center to CDP Private Cloud Base (‘CDP PvC Base’).  Additionally, we’ve released two minor updates, versions 7.1.2 and 7.1.3.  Let’s talk through these changes.  Details:CDP Private Cloud consists of CDP PvC Base and CDP PvC Experiences, which host the decoupled storage and compute layers respectively.  Additionally, CDP PvC Base also hosts Data Hub services for traditional workloads.  CDP PvC Base continues to be the unified platform for CDH and HDP customers to migrate their on-premises workloads to CDP.  Learn more about CDP Private Cloud here.  Key features and improvements available with CDP PvC Base 7.1.3 include:General availability of Ozone Object Store.  This introduces vastly improved storage scaling capabilities over HDFS, addressing performance degradation and fsimage corruption issues, enhanced compatibility, and suppor

We are happy to announce General Availability of CDS 3.0 Powered By Apache Spark 3.0.1. You can download the parcel and apply it directly to provisioned clusters without disrupting your currently running Spark workloads, while taking advantage of all new features and benefits that come with Spark 3.0. This component is generally available and is supported on CDP Private Cloud Base clusters running version 7.1.3 and above. What's New in CDS 3.0? Support of JDK11, Scala 2.12, Python 3.4+ (Python 2.7+ deprecated)Adaptive execution of Spark SQLDynamic Partition PruningBinary files data sourceDataSource V2 Improvements (Pluggable catalog integration)Structured Streaming UIAuto discover and schedule tasks on nodes with GPUs on a YARN clusterKafka connector delegation token (0.10+) See documentation for details and installation. --------------Want to become a pro Spark user?  Sign up for Apache Spark Training.

We are pleased to announce the release of Cloudera JDBC 2.6.18 driver for Apache Impala.  The release has the following enhancements and fixes: Enhancements and New Features [IMPJ-561] The driver now supports Java 11. [IMPJ-566][IMPJ-580] Updated third-party libraries The JDBC 4.2 driver has been updated to use the following libraries: - log4j 2.13.3 - slf4j 1.7.30The JDBC 4.0 and 4.1 versions of the driver continue to use the previous versions of these libraries.[00221973][IMPJ-569]  The driver now supports the values set via the Statement.setFetchSize API call. Resolved Issues[00195066][IMPJ-562] The driver converts UPSERT-SELECT to INSERT-SELECT.[00200861][IMPJ-568] When Impala analytic functions are used, the driver does not honor upper-case column aliases.[00206440][IMPJ-576][IMPJ-574] The driver logs incorrect socket timeout values.[00206060][IMPJ-578] When the UseNativeQuery property is set to 1 and there is a  space between "com

  Cloudera Data Engineering is an integrated, purpose-built experience for data engineers. It delivers a streamlined service for scheduling, monitoring, debugging, and promoting data pipelines quickly & securely across the enterprise at scale.   The key to the experience is a centralized interface that simplifies the job management life cycle from scheduling, deploying, monitoring, debugging, and promotion which alleviate many of the challenges with running Spark jobs in production at scale. Similar to CML and CDW,  CDE is cloud native leveraging Kubernetes where Platform admins can quickly provision virtual compute clusters with strong isolation, capacity auto-scaling and quotas for cost management.   For Platform Admins: Managed Spark Service running on Kubernetes with mixed-version spark deployments accelerating DE workflows with zero setup.  One click provisioning of new workloads with guardrails for CPU and Memory. Data Governance and management throu

We are pleased to announce the release of the Cloudera ODBC 2.6.9 Driver for Apache Hive. The release has the following resolved issue. Enhancements and New Features[00211293][HAR-1156] Updated HiveServer2 connection The driver now supports the Active Passive HS2 Interactive HA mode. For more information, see the Installation and Configuration Guide. [HAR-1080] macOS X notarization The driver now supports notarization on macOS versions 10.14.6 and later.[HAR-1144] CDH 6.3 support The driver now supports CDH 6.3. [HAR-1154] CDP 7.0 support The driver now supports CDP 7.0.Resolved Issues[00147018][HAR-1053] On AIX, when using a different OpenSSL version, the driver terminates unexpectedly. [00220828][HAR-1158] When the SQLBindCol property is called with a NULL pointer, the driver throws an error. [HAR-1045] In some cases, when attempting to continue an interrupted fetch operation after losing contact with the server, the driver might return result sets with missi

We are pleased to announce the release of Cloudera ODBC 2.6.11 driver for Apache Impala.  The release has the following enhancements and fixes: Enhancements and New Features[IMP-659][00143882] Support for DML result set count The driver now reports the correct row count for DML results in server versions that return the correct row count information. Previously, the driver returned "-1". [IMP-783] Server-side properties configuration You can now configure server-side properties as driver-wide properties via the registry or the.impalaodbc.ini file. [IMP-786] Support for CDP 7.0 The driver now supports CDP 7.0. Resolved Issues [IMP-662] [00173663] When an incorrect user ID or password is entered in SASL, the driver returns the error message "Error from the Impala Thrift API: No more data to read." This issue has been resolved. The driver now returns a more detailed error message. [IMP-691][00146553] The driver returns string data untruncated regardless

We're pleased to announce the release of Cloudera Data Center (DC) 7.1. CDH and HDP customers are encouraged to upgrade to CDP DC 7.1 for improved enterprise data management capabilities and new platform innovations.    This release offers both CDH and HDP customers an upgrade path to CDP DC (and to CDP Private Cloud when available) and brings Cloudera Runtime 7.1, the best of CDH and HDP to our on-premise customer base.    Key features and improvements available in DC 7.1 include: Streams Messaging: Complete and comprehensive Kafka streaming experience improving operational efficiency, business continuity and scalability. Data Engineering:  Improved performance and interoperability for Apache Spark and management of data engineering workflows and pipeline creation. Data Warehouse:  Faster SQL analytics on larger data sets, deeper understanding from unstructured data sources, and easier visualizations of business insights. Machine Learning:  Dat

We are pleased to announce the release of Cloudera JDBC 2.6.17 driver for Apache Impala.  The release has the following enhancements and fixes: Enhancements and New Features[IMPJ-547] The driver now supports CDH versions 5.11 through 5.16 and 6.0 through 6.3.[IMPJ-493] The driver now uses version 2.10.1 of the Jackson library. Previously, the driver used Jackson version 2.9.9. Resolved Issues[00147825][IMPJ-516] The driver does not generate the correct syntax for character parameters in prepared statements if the length of the parameter value is greater than 510.[00147114][IMPJ-512] getParameterMetaData() returns VARCHAR for all parameters.[00147464][IMPJ-514] When you use the com.cloudera.impala.jdbc.DataSource class to connect with the JDBC 4.1 or 4.2 version of the driver, the driver returns a class cast exception.[IMPJ-520] The JDBC 4.1 version of the driver fails to connect to servers that require encryption using TLS 1.2.  This issue has been resolved. However

We are pleased to announce the release of Cloudera JDBC 2.6.10 driver for Apache Hive  The release has the following fixes and enhancements: Enhancements and New Features[00176863][HIVEJ-584] Updated Thrift library The JDBC 4.2 version of the driver now uses version 0.13.0 of the Thrift library. Previously, this version of the driver used Thrift version 0.12.0. The JDBC 4.0 and 4.1 versions of the driver continue to use Thrift version 0.12.0. [HIVEJ-575] Updated Jackson library The driver now uses version 2.10.1 of the Jackson library. Previously, the driver used Jackson version 2.9.9. Resolved Issues[HIVEJ-577] The JDBC 4.1 version of the driver fails to connect to servers that require encryption using TLS 1.2. [HIVEJ-578] When you invoke the JDBC 4.2 driver with the Datasource class, an exception occurs. [00176503][HIVEJ-583] When you use the driver with the Denodo application, it returns the following error: "Could not initialize Class". [HIVEJ-596

Just Released: Cloudera AI Workbench With Support for CDP Data Center CDSW 1.7.2 is now available, bringing support for CLoudera Data Platform Data Center Edition 7.0 (CDP-DC) and improved usability for experiments and model deployment. CDSW on CDP-DC enables an improved best of breed data science platform experience for your data science teams on our latest enterprise data platform.   Whether you’ve already begun to migrate to CDP Data Center or are just starting your journey to the cloud, CDSW on CDP-DC enables continuity in your data science experience and a more frictionless path to Cloudera Machine Learning (CML) — Our enterprise cloud-native data science and machine learning platform for CDP Private Cloud and Public Cloud.   Support for virtual clusters in CDP Data Center Customers using CDP-DC as a stepping stone in their journey to Private Cloud can now reduce impact to other workflows and noisy neighbors by leveraging CDSW in isolated virtual clusters with assigned r

We are pleased to announce the release of Cloudera ODBC 2.6.9 driver for Apache Impala.  The release has the following fixes and enhancements: Enhancements and New Features [IMP-704][00174819] Improved performance with multiple concurrent connections.  The driver now retrieves data much faster in scenarios that involve multiple concurrent connections to the data store. This improves performance when working with applications such as Power BI. [IMP-696][IMP-708][IMP-722] Updated third-party libraries. The driver has been updated to use the following libraries: Boost 1.64.0 (previously 1.57.0) libcurl 7.66.0 (previously 7.60.0) Expat 2.2.9 (previously 2.2.0) OpenSSL 1.1.1d (previously 1.1.0j) Resolved Issues [IMP-699][00173163] In some cases, when the driver submits a query that contains a Boolean value expression, the driver returns an "Unsupported query" error.   [IMP-724] The driver terminates unexpectedly when both of the following occur:  Multiple d

Announcing Cloudera Enterprise 6.3.3 We are pleased to announce the release of Cloudera Enterprise 6.3.3, the world’s leading data platform for machine learning and analytics, optimized for the cloud. This maintenance release includes all the features and capabilities of the previous minor release (Cloudera Enterprise 6.3) with the addition of numerous  important bug fixes.  Also note the current list of known issues with Cloudera Enterprise 6.  We also encourage you to review the new Upgrade Guide that now includes the ability to create customized upgrade instructions based on your specific environment and upgrade path. New Features & Capabilities Cloudera Enterprise 6.3.3 now supports running on RedHat Enterprise Linux 7.7 and CentOS 7.7Cloudera Manager provides new tuning options to increase the scalability of monitoring services for large CDH deployments Important Note About Installation Beginning with Cloudera Manager 6.3.3 and CDH

Greetings!   We are pleased to announce General Availability of HDP-3.1.5.0 and AMBARI-2.7.5.0!   With this release HDP is supporting single catalog for Hive/Spark and Hive table-level replication. To keep up with the increased customer interest, we delivered HBase 2.1.6 and GCS Connector 1.9.17 as well.   We have also delivered nearly 200 fixes to increase stability, usability, supportability and performance, enhance security and improve the overall customer experience with HDP and Ambari.   A big kudos and a mountain of gratitude to each and every member of the sustaining, development, quality, release engineering, documentation, biz ops and legal team, to deliver this release with such high quality.   Documentation Download   Thank you all again for your continued support and commitment on this journey!   As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You ca

On November 22, 2019, we released CDP Data Center, our next-generation on-premises offering, representing the strongest elements of the former Cloudera and Hortonworks platforms accompanied by new innovations for better scalability, performance and private cloud readiness. Relative to Cloudera’s Enterprise Data Hub, CDP Data Center now offers:Better security:  attribute-based, fine-grained access controls featuring dynamic row filtering and column masking with Ranger. Better governance: more scalable data lineage, data discovery, and metadata search with Atlas 2.0.Better performance:  faster Hive queries and support for ACID transactions. Relative to Hortonworks Data Platform, CDP Data Center brings:Better management:  manage multiple clusters from one control plane, one-click wire encryption, and streamlined maintenance operations with Cloudera Manager.Better SQL analytics: low-latency interactive SQL with Impala, Kudu, and Hue.Better encryption: enterpri