Announcements

COD now supports HDFS for fast and consistent performance.Cloud Object Storage has a high variance in latency, and sometimes the delay in accessing the data is high. The operational cost in managing such an operational database is expensive, as it reads millions of data per second. COD now deploys HDFS on Cloud Block Storage, which ensures that the performance characteristics are similar to on-prem. This feature simplifies the lift-and-shift efforts to the cloud.This capability includes an updated ‘update-database’ command in CDP CLI Beta to enable tuning of auto-scaling based on HDFS utilisation. This is available starting in COD v1.13 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

Cloudera now supports Phoenix access through the Apache Hue SQL assistant.  You can now access Phoenix and query the data from the Hue web UI out of the box and leverage Hue’s rich SQL support (for example, type ahead) to query Phoenix for operational insights or developer testing of SQL statements. This is available starting in COD v1.11 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

Cloudera is aware of CVE-2021-4104, which affects the Apache Log4j 1.x JMSAppender. This flaw only affects software that is explicitly configured to use the JMSAppender, which is not the default, or when the attacker has write access to the Log4j configuration for adding JMSAppender. Cloudera does not use JMSAppender in its products and it is not used by default in log4j properties. Cloudera customers do not need to take any action to address CVE-2021-4104. If you have further questions, please contact Cloudera Support through our My Cloudera Support portal.   Because Cloudera Manager and Ambari allow authenticated users with privileged access to modify cluster configuration to insert custom logging configuration, customers are advised to review the logging configuration for their clusters to ensure that they do not contain references to the JMSAppender. To find these settings, look for the following based on the cluster management tool in use: Cloudera Manager: "{SERVICE_NAME} L

With CML's multi-version Spark support, CML users can now run different versions of Spark side by side, even within a single project. Users can select to use Spark 3 in the most recent CML version and take advantage of performance and stability improvements in the latest version of Spark.  Data Scientists can run workloads in both Spark 2 and Spark 3 within the same CML Workspace, thus maintaining backward compatibility with existing workloads while developing new applications on the latest version of Spark. Users can select the Spark version they want to use for each workload, making it easy to migrate older jobs using Spark 2 to Spark 3. 

The latest release of Cloudera DataFlow for the Public Cloud (CDF-PC) introduces several new features allowing more Apache NiFi users to run their data flows on CDF-PC.  This includes:The ability to deploy NiFi flows which require custom processors / custom controller services[Tech Preview] The ability to use CDF-PC on AzureSupport for AWS setups that use non-transparent proxy servers for outgoing network communicationSupport for private AKS/EKS clustersReduced infrastructure cost through optimized instance use and NiFi deployment sizesLearn more about the new features in the CDF-PC documentation and take an interactive tour of CDF-PC

As a courtesy, we are publishing a quarterly update as a reminder of upcoming End of Support (EOS) dates. Our Knowledge Hub provides material to assist with moving to CDP from operational efforts to application changes.   The following products will reach their EOS date by or before January 31, 2022 (as more specifically listed below): Cloudera Streaming Analytics (CSA) 1.1 - January 2022 Cloudera Enterprise Products (CDH) 6.1 - December  2021 Hortonworks Data Platform (HDP) 3.1 December 2021 Hortonworks Data Lifecycle Manager (DLM) 1.3 December 2021 Hortonworks Data Steward Studio (DSS) 1.3 December 2021 Hortonworks Data Analytics Studio (DAS) 1.1 November 2021 1.2 December 2021 Cloudera HDP Search  5.0 - December 2021 Cloudbreak  All Versions - December 2021 The following Products will reach their EOS date in the next 12 months: Cloudera Data Science Workbench 1.5 - 1.6 March 2022 1.7 - 1.8 March 2022 1.9 - June 2022 CD

 We are pleased to announce the release of Cloudera JDBC 2.6.24 driver for Impala.  This release has the following enhancements and fixes: Enhancements and New Features[IMPJ-630] Ignore transactions support[IMPJ-633][IMPJ-655] Updated third-party libraries Resolved Issues[00300232][IMPJ-648] When executing INSERT queries, the connector casts empty strings to CHAR(1)[IMPJ-652] The ImpalaJDBCDataEngine does not check getFunctions and returns an empty result set[IMPJ-653] In some cases, when using Safari, SSO authentication fails and returns a 302 error. Workflow Changes[IMPJ-661] Removed third-party libraries Getting Started with the Cloudera DriverRead the Cloudera JDBC Connector 2.6.24 for Impala Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs t

COD provides the ability to disable the ephemeral storage using a new command ‘--disable-ephemeral-storage’. This is available starting in COD v1.15 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD now supports a built-in coprocessor called AggregateImplementation that facilitates aggregation function computations (min, max, sum,avg, median, std) at the region level. This yields better performance because you need not get all the data to perform these calculations. 'AggregateImplementation' is enabled by default, and you can use the AggregationClient in HBase to perform RegionServer side aggregation, for example, row count. This is available starting in COD v1.16 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD now supports an improved auto-scaling algorithm that considers the latency of the user operations. COD now prioritises user operations over system operations that results in a reduced cost of infrastructure with a minor increase in replication. This is available starting in COD v1.16 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed). 

COD supports listing all the HBase snapshots created within a database of an environment using these filters, time range, snapshot name, and command ID. For example, you can use the time range filter to obtain the snapshots created within a particular time, the command ID filter to know the status of a snapshot operation. This is available starting in COD v1.15 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD introduces a technical preview version of Multi-AZ deployment capability and is available by special request through your account team.Once enabled, COD, by default, deploys all databases across 3 availability zones identified in the SDX by the network subnets that were provided. COD CLI beta provides a new command ‘--disable-multi-az’ to limit deployments to a single availability zone. This is available starting in COD v1.15 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

With the launch of the Cloudera Public Cloud 7.2.12, the Streams Messaging for Data Hub deployments have gotten some interesting new features! From this release, Streams Messaging templates will support scaling with automatic rebalancing allowing you to grow or shrink your Apache Kafka cluster based on demand. Another notable item is that Streams Replication Manager (SRM) will now support multi-cluster monitoring patterns and aggregate replication metrics from multiple SRM deployments into a single viewable location in Streams Messaging Manager (SMM.) Last but not least is Apache Atlas and Schema Registry (SR) Integration, now you will be able to view Kafka topic schemas in Atlas letting you navigate data lineage from consumers and producers and see the schema of the topic they use without having to navigate back to the SR UI. Learn more about these features in our CSM 7.2.12 release blog!https://blog.cloudera.com/new-features-in-cloudera-streams-messaging-public-cloud-7-2-12/

We are pleased to announce the release of CSA Community Edition (CE). You can install and run CSA 1.5 directly from your desktop or other development nodes. Using Docker Compose, you can easily run a stand-alone Cloudera Streaming Analytics stack on your laptop or any other development node. The only pre-requisite is Docker. Head over to the Cloudera downloads page to get the docker-compose YAML file, and check out the documentation to get started in just a few minutes. There is an included tutorial to get your first stream processor built and running. You can read more about the release in this blog post.

We are pleased to announce the release of Cloudera ODBC 2.6.14 driver for Impala.  This release has the following enhancements and fixes: Enhancements and New Features[IMP-854] SSOIgnoreDriverNoPrompt support [IMP-857] Updated OpenSSL support Resolved Issues[IMP-852][00303650] The cyrus-sasl libraries do not pass the Binskim scan Known Issues[IMP-843][00294390] Limited support for query translation  Workflow Changes[IMP-717][IMP-719] Removed support for Visual Studio 2013[IMP-853][IMP-860][IMP-879][00305355][00352154] Updated HTTPAuthCookies default value Getting Started with the Cloudera DriverRead the Cloudera Impala ODBC 2.6.14 Driver for  Impala Release Notes and Installation Guide. Download the connector from the Cloudera Connectors page.  As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our ext

With Cloudera Data Engineering (CDE) Pipeline authoring UI, any CDE user irrespective of their level of Airflow expertise can create multi-step pipelines with combination of out of the box operators (CDEOperator, CDWOperator, BashOperator, PythonOperator). More advanced users can still continue to deploy their own custom Airflow DAGs (Directed Acyclic Graphs) as before, or use the Pipeline authoring UI to bootstrap their projects for further customization. And once the pipeline has been developed thru the UI, they are deployed and operationally managed through the same best in class APIs and job life-cycle management the users have come to expert from CDE. Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex pipelines. Until now, the setup of such pipelines still required knowledge of Airflow and the associated python configuration. As a result users tended to li

Cloudera Data Engineering (CDE) in Cloudera on cloud introduces a new bin-packing auto-scaling policy for more efficient resource allocation and cost reduction.  Customers with bursty data pipelines with overlapping  schedules can reap the cost benefits of faster scale down through the use of the new bin-packing policy.  Bin-packing policy is appropriate in situations where there's a mix of regular and bursty workloads deployed with some scheduling overlap. Usually, as nodes scale to meet the demand of a workload burst, there would be slow scale down because new jobs would be distributed across the new nodes preventing them from being freed up.   Although this helps to reduce host spots, it leads to underutilization across the large number of nodes.  With bin-packing as new jobs come online they are more efficiently allocated (ie - "bin-packed") to a smaller subset of nodes freeing the majority of the new nodes that were added. This fea

As a key component of Cloudera Data Engineering (CDE), Apache Airflow has served as a flexible orchestration service for data engineers and practitioners to develop and operationalize end-to-end data pipelines. Today, many customers use the managed Airflow service to avoid the administrative overhead of maintaining and tuning their own full-fledged scheduler. Instead they choose to rely on the out-of-the-box security and autoscaling compute enabled in CDE to deploy tens to hundreds of DAGs (Directed Acyclic Graphs) using CDE's job management APIs. And with integration with CDP data services, pipelines can flexibly tap into the efficient containerized compute of Spark in CDE and Hive in Cloudera Data Warehouse (CDW).With Airflow 2.1 as the new default managed scheduler, customers can continue to rely on the low administrative overhead they have come to expect while users can reap the benefits of the latest developments in the upstream community. As any major release, many aspe

We are excited to announce the general availability of Cloudera Streaming Analytics (CSA) 1.5.0 on CDP Private Cloud Base. This release includes a massive set of updates to SQL Stream Builder (SSB) as well as an update to Flink 1.13. This release Includes a RESTful API interface, first-class DDL support, and GA support for change data capture (CDC) right in SSB!Some of the new features are:Multi-line DDL support in the editor. Users can now enter CREATE TABLE or INSERT statements inside the main editor - it supports SET statements, multiple operations at once, and even multiple INSERT statements at once.CDC support is now GA. This allows customers to wire up Debezium based change data capture to popular databases. No other systems and components are needed!A RESTful API. Customers wanted a way to store SQL in git or on the file system and then start jobs via the command line. CSA 1.5 includes our first pass at this API!Check out this blog post and the documentation for more inform

The latest release of Cloudera DataFlow for the Public Cloud (CDF-PC) introduces the ability to manage flow definitions in the catalog and create flow deployments using the CDP Beta CLI. This allows users to fully automate the flow deployment process and integrate it into their CI/CD pipelines. In addition to new CLI features, this release also introduces the following new features:A new ReadyFlow is available to move data between S3 buckets leveraging S3 bucket notifications.DataFlow now supports Cloudera RAZ (fine-grained access control) for object store access.The Dashboard, Catalog, and Environments pages now persist search queries and applied filters for a better user experience.When deploying a flow definition, you can now select whether the NiFi flow should be started automatically.Learn more about the new features in the CDF-PC documentation and read our detailed Blog Post to automate flow deployments

We're pleased to announce the release of Cloudera Private Cloud (PvC) Data Services 1.3, which delivers key new features and improvements across the platform, including Embedded Container Service (ECS) and Cloudera Data Engineering as an additional service.In this release, we’ve added the following new platform capabilities:ECS: The new capability enables you to run CDP Private Cloud Data Services directly on bare metal or virtual machines by automating the installation and management of all container-related dependencies. ECS greatly simplifies deploying a private cloud and Data Services, providing the familiar Cloudera Manager as a single pane of glass to manage the full deployment. With ECS, customers no longer need to worry or learn about Kubernetes and related technologies to get the benefits of containerized analytic applications running on-premises. Support for CDP CLI: Perform the same actions as can be performed from the Management Console through the CLI, letting you automate

COD provides a new option ‘--os-upgrade-only’ for the ‘upgrade-database’ command in CDP CLI, which you can use to upgrade the operating system to the latest supported version addressing any CVEs that were patched in the OS. Running the ‘upgrade-database’ command with this option does not affect the CDP runtime version running on the given cluster.For more information, see CDP COD CLI Beta command reference. This is available starting in COD v1.15 Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

In June 2021, Apache Sqoop was retired and moved to the Apache Attic. While Sqoop will no longer be maintained at Apache, Cloudera CDP Public and Private Cloud customers can still expect full support including patches, hotfixes and prompt consideration of enhancement requests. For more information, please see the following resources:Migrating Data Using Sqoop in CDP Public CloudApache Sqoop changes after upgrading from CDH to CDP Private Cloud BaseApache Sqoop's apache.org pageApache Software Foundation's Board Resolution Terminating the Apache Sqoop Project 16 June 2021Apache Sqoop in the Apache Attic