Announcements

You can now define custom recipes while creating an operational database using the --recipes option in CDP CLI Beta. You can register pre-created recipes during database creation based on the instance groups in your database. The recipes are executed automatically for the specified nodes based on the recipes’ type.Use the following example command to define custom recipes for your operational database. cdp opdb create-database --environment-name <ENVIRONMENT_NAME> --database-name <DATABASE_NAME> --recipe names=<rec1,rec2>,instanceGroup=<MASTER> names=<rec2>,instanceGroup=WORKER names=<rec3,rec4>,instanceGroup=<GATEWAY> To know more about the recipes, see Recipes.To know more about the --recipes option, see CDP CLI Beta. This is available starting in COD v1.32 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (n

We are excited to announce the general availability of Cloudera Streaming Analytics (CSA) 1.10.0 on CDP Private Cloud Base. This release includes a massive set of improvements to SQL Stream Builder (SSB), including the addition of built-in widgets for data visualization, as well as a rebase to Flink 1.16. Some of the key features of this release are: Rebase to Apache Flink 1.16 - Apache Flink 1.16 is now supported in CSA 1.10.PyFlink Support - The Python API for Flink is now supported in CSA. Customers can now create Flink DataStream applications using Python, besides Java and Scala, to build scalable batch and streaming workloads like real-time data processing pipelines, large-scale exploratory data analysis, Machine Learning (ML) pipelines and ETL processesBuilt-in Widgets for Data Visualization - Built-in data visualization widgets have been added to the SQL Stream Builder (SSB) UI to provide a quick and simple way to visualize data from streaming jobs and materialized vie

The CDP Public Cloud Release Summary for the month of June is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. Seehttps://docs.cloudera.com/cdp-public-cloud/cloud/release-summaries/topics/announcement-202306.html 

SummaryCloudera Data Engineering (CDE) 1.19 introduces interactive Spark sessions for development workflows to take advantage of autoscaling compute and orchestration capabilities that's hybrid  and multi-cloud ready. Since there is no one size fits all approach to development, CDE interactive sessions give data engineers flexible end-points to start developing Spark applications from anywhere -- in a web-based terminal, local CLI, favorite IDE, and even via JDBC from third-party tools. CDE exposes sessions as first-class entities via the APIs, as well as the UI and CLI, allowing users to navigate seamlessly across interfaces. For example, initiate the session through the UI, start interacting with it in the web-based shell, then drop into your local terminal for a spark-shell experience.Interactive Sessions Video  Complete Feature List: Interactive Sessions (Tech Preview) Both CLI and web based interactive shell sessions are now supported. 

The latest release (2.5.0-b210) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces NiFi 1.21, flow metrics driven auto-scaling, advanced UIs for JoltTransformJSON and UpdateAttribute processors, bulk actions in the Flow Designer, UDR support on Azure and makes in-place upgrades generally available to all customers. Flow Deployments and Test Sessions now support the latest Apache NiFi 1.21 release.In-place upgrades are now generally available to all customers. For more information, see Service upgrade.Flow Deployments now support Flow Metrics Scaling in addition to CPU utilization based auto-scaling. For more information, see Auto-scaling flow deploymentsWhen selecting Private Cluster during enablement on Azure, CDF no longer provisions a public load balancer and now supports User Defined Routes (UDR).Users can now provide their own values for the Kubernetes Pod CIDR Range and Kubernetes Service CIDR Range during enablementCDF now inherits load balancer configuration includ

On the COD UI, when you create an operational database the scale option Medium Duty is renamed to Heavy Duty. For more information, see Creating a database using COD. This is available starting in COD v1.32 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

In CDP CLI, when you select the --scale-type option as HEAVY, COD allocates larger SSD storage (for example, gp2 on AWS, StandardSSD_LRS on Azure, or pd-ssd on GCP) for both master and leader node types. For more information, see CDP CLI Beta. This is available starting in COD v1.32 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

 We're thrilled to announce the release of Cloudera Private Cloud (PVC) Data Services 1.5.1, the next iteration of Cloudera’s on-premises, cloud-native open data lakehouse, bringing the power of cloud-native data apps to your data center.  This release is jammed with 80+ new features across the platform and data services - Cloudera Data Engineering, Cloudera Data Warehouse, and Cloudera AI - to deliver faster insights on a more robust, secure, and performant platform.  Each of these are covered in detail in the release notes.   Some key platform highlights of this release include: Object storage with Cloudera Object Store (powered by Apache Ozone) is now GA on all data services, enabling modern data use-cases faster performance, denser storage, and better scalability.Open data format with Apache Iceberg V2 is available as a tech preview on all three data services, for greater flexibility and interoperability with your private open data lakehouse.

Here are some highlights from the month of May   COMING NEXT! Streaming Data Pipeline Development Check out the FY24 Cloudera Meetup Events Calendar for upcoming & past event details!   182 new support questions 14 new community articles 472 new members Rank  Community Article Author Components/ Labels #1 Get data from Oracle by Apache NiFi , then save to Hive Parquet External Table Zhen Zeng @zzeng  Apache NiFi #2 Spark3 Integration Examples Ranga Reddy @RangaReddy  Apache HBase Apache Hive Apache Iceberg Apache Kafka Apache Kudu Apache Ozone Apache Phoenix Apache Spark Cloudera Data Platform (CDP) #3 Operationalize NiFi data flows with Cloudera DataFlow Steven Matison @steven-matison  Apache Spark Cloudera Data Platform (CDP) #4 CDW - RAZ Enabled : How to add S3 Bucket in Two Steps (That was EASY) Ryan Cicak@RyanCicak  Apache Ranger Cloudera Data Warehouse (CDW) #5

The CDP Public Cloud Release Summary for the month of May is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. Seehttps://docs.cloudera.com/cdp-public-cloud/cloud/release-summaries/topics/announcement-202305.html 

In CDP CLI, the --scale-type option now supports all the three options --scale-type (string) <MICRO, LIGHT, HEAVY>. COD has added support for additional parameters, LIGHT and HEAVY. You can specify these options for the --master-node-type.--scale-type LIGHT (-master-node-type LITE)--scale-type HEAVY (-master-node-type HEAVY)Additionally, COD has removed the --master-node-type (string) <LITE,HEAVY> CDP CLI option.For more information, see CDP CLI Beta. This is available starting in COD v1.31 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD now allows you to create a smaller cluster with one Gateway node and one Worker node using a new scale type Micro Duty while creating an operational database through COD UI.The Micro database is a two node cluster in which the Gateway node processes the activities of the Master and Leader nodes, thereby removing the need of these nodes. You can use a Micro database for testing and development purposes.For more information, see Creating a database using COD. This is available starting in COD v1.30 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD now allows you to vertically scale up the COD clusters from a Light Duty to a Medium Duty instance type. You can upgrade the instance type of a COD cluster that belongs to a Master or Gateway node. To know more about the vertical scaling, see Vertical scaling COD clusters. This is available starting in COD v1.30 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

The CDP Public Cloud Release Summary for the month of April is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. Seehttps://docs.cloudera.com/cdp-public-cloud/cloud/release-summaries/topics/announcement-202304.html 

Data ingestion is a common task in the data science workflow, which often involves coordinating with multiple teams. With the "Add Data" action on CDP Data Connections, data scientists can now easily upload data into CDP Data Stores such as Impala or Hive Virtual Warehouses to manage and govern data at scale. This means that data scientists can focus on analyzing and working with their own data rather than dealing with the complexities of data ingestion.  To get started with this feature, users can simply open the “Data” tab in their CML Project and click on the "Add Data" action on the CDP Data Connection they wish to use, and follow the prompts to upload their data into a CDP Data Store.   In addition to simplifying the data ingestion process, the "Add Data" action also provides users with several options for customizing the data import. These options include selecting the database and table name for the data, as well as selecting the column delimiter and loc

We are excited to announce the launch of Custom Connection Support in Cloudera Cloudera AI (CML) that enables data scientists to seamlessly connect to external data stores from within CML. This feature helps data scientists discover all of their data independently, without worrying about implementation and connectivity details, unlocking their machine learning use cases from the get-go. Many users struggle with accessing data from various sources, such as legacy on-prem databases (Oracle, MSSQL, MySQL), serverless cloud databases (Redshift, Snowflake, SAP HANA Cloud, BigQuery), APIs, and specialized data stores (Neo4j). Until now, CDP administrators set up data replication or ingestion pipelines, to make data discoverable and accessible within CDP or work directly with data scientists to provide the necessary endpoints, configurations, and authentication details to set up connections manually. This process not only delayed machine learning use cases but also burdened CDP administr

Here are some highlights from the month of March   COMING NEXT! Streaming Data Pipeline Development   Check out the FY24 Cloudera Meetup Events Calendar for upcoming & past event details!     198 new support questions 2 new community articles 592 new members Rank  Community Article Author Components/ Labels #1 Processing files within the CML EFS (better I/O performance) - Edge Case with non-tabular data Ryan Cicak @RyanCicak   Cloudera Machine Learning (CML)   We would like to recognize the below community members and employees for their efforts over the last month to provide community solutions.    See all our top participants at Top Solution Authors leaderboard and all the other leaderboards on our Leaderboards and Badges page.     @MattWho @smdas @steven-matison @mszurap @ckumar @cotopaul @SAMSAL @davehkd @BrianChan @jorgeoa

The CDP Public Cloud Release Summary for the month of March is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. Seehttps://docs.cloudera.com/cdp-public-cloud/cloud/release-summaries/topics/announcement-202303.html 

We are pleased to announce the release of Cloudera ODBC 2.6.16 driver for Apache Hive.This release has the following fixes and enhancements: Resolved Issues[HAR-1335] In some cases, when handling primary keys and foreign keys catalog functions, the connector terminates unexpectedly. Known Issues[HAR-1171] The Hive server does not currently support using scientific notation for FLOAT type. As a workaround, to use scientific notation, cast the data to a DECIMAL Getting Started with the Cloudera DriverRead the Cloudera ODBC Driver 2.6.16 for Apache Hive Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

  Visit the Future of Data Meetup group page to join a local group & register for events!   178 new support questions 7 new community articles 517 new members   Community Article Author Components/ Labels Open Data Lakehouse powered by Apache Iceberg on Apache Ozone Saketa Chalamchala@saketa  Apache Hive Apache Impala Apache Spark Cloudera Data Platform (CDP) Cloudera Data Platform Private Cloud (CDP-Private) Installing Django in Cloudera Machine Learning (CML) Ryan Cicak@RyanCicak  Cloudera Machine Learning (CML) How to Spark Roll Event Log Files in CDP Ranga Reddy@RangaReddy  Apache Spark Cloudera Data Platform (CDP) CDSW - CML || How to import a private repository from Github, gitlab Girish Vaggar @gireeshn  Cloudera Data Science Workbench (CDSW)  Cloudera Machine Learning (CML) Navigator Metadata Service is not starting in CDH6.3.4 Amararam Gehlot @agehlot  Cl

The latest release (2.4.0--b317) of Cloudera DataFlow (CDF) on CDP Public Cloud makes the Flow Designer generally available to all CDP Public Cloud customers, introduces a streamlined way to collect log files, supports new NiFi and Kubernetes versions and introduces several usability improvements when creating and managing flow deployments through the UI. The ability to design flows with the new Flow Designer is now generally available to all CDP Public Cloud customers.The CDP cli now supports commands to generate diagnostic bundles for CDF for troubleshootingFlow Deployments and Test Sessions now support the latest Apache NiFi 1.20 releaseWhen creating a deployment through the deployment wizard, users can now filter for sensitive or empty parametersCDF now creates Kubernetes clusters with version 1.23 in AWS EKS and 1.24 in Azure AKS.Upgrading and restarting deployments now triggers a NiFi Health Alert if the NiFi cluster does not return to a healthy state. This can be cleared by

On the COD UI, you can find the JWT configuration details under <Database> > Connect > HBase Client Tarball > JWT Configuration. You can refer to these configurations to set up a connection to HBase with a JWT token and build your own truststore JKS file.For more information, see Configuring JWT authentication for HBase client. This is available starting in COD v1.28 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD UI now allows you to select the storage type when creating an operational database. You can either select Cloud Storage or HDFS.Earlier, if you had to use the HDFS as the storage type, you were required to use the --use-hdfs option on COD CLI while creating the operational database. Now COD UI is enhanced to let you select the storage type when you create an operational database.For more information, see Creating a database using COD. This is available starting in COD v1.28 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

The Cloudera Data in Motion (DiM) team is pleased to announce the general availability Cloudera Streaming Analytics (CSA) 1.9.0 on CDP Private Cloud Base 7.1.7 SP2 and 7.1.8. This release includes a massive set of improvements to SQL Stream Builder (SSB) as well as updates to Flink 1.15.1. These changes are focused on enhancing the user experience and removing objections and blockers in the sales cycle. All the features described below are already available in the Cloudera Stream Processing - Community Edition release, which is the fasted way for you to try them out for free. Links:DocumentationRelease notesCSP Community Edition Download and InstallBlog - A UI That Makes You Want To StreamBlog - SQL Stream Builder Data TransformationsBlog - Job Notifications in SQL Stream Builder Key features for this release: Reworked Streaming SQL Console: The User Interface (UI) of SQL Stream Builder (SSB), the Streaming SQL Console has been reworked with new design elements. &nb

Cloudera AI and data science libraries and frameworks have grown at an exponential pace along with algorithmic advancements with the introduction and evolution of Neural Networks to Transformer libraries. To keep up with this innovation, our customers have always asked for a pluggable architecture where the libraries can be chosen and hand-selected by their users, yet works within the ML platform that is enabled by CML from data exploration to model operations. Open source ML Runtimes afford that extensibility to our partners and customers. They can extend and create purpose-built runtimes for data science teams and projects. With our prior investment in the PBJ (Powered by Jupyter) architecture, we can now rely on open source, community-supported protocols and release a new family of our ML Runtimes to better align with the Jupyter ecosystem. With this rebuilt infrastructure, customers and partners will no longer need to build runtime images starting from Cloudera base images. Th