Announcements

Here are some highlights from the month of January 194 new support questions 5 new community articles 573 new members     Rank  Community Article Author Components/ Labels #1 CDP Public Cloud DL /DH vertical scaling for Azure based cluster  Usman Khan  @usman35  Cloudera Data Engineering (CDE) Cloudera Enterprise Data Hub Cloudera Machine Learning (CML) Data Engineering UI #2 How to install l Apache NiFi on Macbook, for learning Zhen Zeng  @zzeng  Apache NiFi #3 Importing additional python modules while making use of ExecuteProcess processor in NiFi Manick Mehra  @mmehra  Apache NiFi #4 Vertical scaling of DataHub Master VM in AWS Usman Khan @usman35  Cloudera Data Engineering (CDE) Cloudera Data Platform (CDP) Cloudera Data Science and Engineering Cloudera Enterprise Data Hub Cloudera Machine Learning (CML) #5 NFS Disk migration for Cloudera Machine Learning - Private C

We're excited to announce the launch of the Tech Preview of Cloudera Model Registry, a centralized repository for storing, managing, and deploying machine learning models and their associated metadata. This powerful new tool is designed to simplify the MLOps process, making it easier for organizations to develop, deploy, and maintain machine learning models in a production environment. The Model Registry solves the problem of fragmentation and lack of visibility in MLOps workflows. It provides a single source of truth for model versions and lineage, enabling organizations to streamline their workflows and enhance the traceability and reproducibility of model development. The new Model Registry is now available in CML Private Cloud 1.5 and in CML Public Cloud. To get started with the Tech Preview version, reach out to us!  You can learn more about the new Model Registry in our documentation.

We're pleased to announce the release of Cloudera Private Cloud Base 7.1.7 Service Pack 2 (SP2) and Cloudera Manager version 7.6.7, which deliver ongoing platform stability and security for the long term supported (LTS) 7.1.7 release.  LTS releases provide a 4 year support roadmap with service packs designed to package cumulative hotfixes and additional updates to deliver continued platform stability and performance. Service Pack 2 provides the following key updates:Enhanced upgrade path to Private Cloud Base (PvC) – CDH 5 & 6 customers as well as HDP 2 & 3 customers can now upgrade to CDP PvC Base 7.1.7 SP2 in place. Rollback procedures are also available to support upgrades from CDH 5 & 6 and HDP 2 & 3.Replacement of log4j1 with reload4j – As part of our commitment to remediate critical security vulnerabilities, we have completely replaced log4j1 with reload4j in CDP PvC Base 7.1.7 SP2. Please see the official Release Notes for the full list of fea

We're pleased to announce the release of Cloudera Private Cloud (PVC) Data Services 1.5.0, which delivers new features and improvements to accelerate your modernization to cloud native architectures on premises.  Some key highlights of this release include: Lower minimum hardware requirements, as well as pathways to convert CDP PVC Base nodes into PVC Data Services nodes. Easier upgrades of platform and data services, including migration automations from PVC Base to PVC Data Services for Cloudera Data Engineering and Cloudera AI.Backup and restore functionality for Control Plane and Cloudera Data Warehouse (CDW)Added Iceberg as a technical preview for CDE and CDW, enabling customers to embrace the first open private data lakehouse on premises.  Additionally, there are numerous improvements to platform security, stability, performance, and of course new features across each of the data services - CDE, CDW, and CML. Please see the Release Notes for CDP

Here are some highlights from the month of December 162 new support questions 1 new community article 493 new members       Rank  Community Article Author Components/ Labels #1 How to Integrate CDE with COD and Reading & Writing to Phoenix Tables using Data Engineering Service by leveraging PySpark on CDP Platform Syed Saleem Hameed @shameed  Apache HBase Apache Phoenix Apache Spark Cloudera Data Engineering (CDE) Cloudera Data Platform (CDP) Cloudera Operational DB   We would like to recognize the below community members and employees for their efforts over the last month to provide community solutions.  See all our top participants at Top Solution Authors leaderboard and all the other leaderboards on our Leaderboards and Badges page.   @smdas @MattWho @Kartik_Agarwal @rki_ @myzard @sanjaysubs @hegdemahendra @jorge_florencio    Share your expertise and answer

 We are pleased to announce the release of Cloudera JDBC Connector 2.6.30 for Impala.  This release has the following enhancements and fixes: Enhancements and New Features [00406889][IMPJ-723][IMPJ-724] Updated Jackson library.The connector now uses the following Jackson versions:jackson-annotations 2.13.4 (previously 2.13.2)jackson-core 2.13.4 (previously 2.13.2)jackson-databind 2.13.4.2 (previously 2.13.2.2) Getting Started with the Cloudera DriverRead the Cloudera JDBC Connector 2.6.30 for Impala Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

COD now provides a CLI option, --enable-region-canary to enable the HBase region canaries while creating an operational database.Use the following command to enable the HBase region canaries.cdp opdb create-database --environment-name ENVIRONMENT_NAME --database-name DATABASE_NAME --enable-region-canaryIf the command is successful, the following HBase canaries are added. You can view them on the Cloudera Manager UI under the HBase service.hbase_region_health_canary_enabledhbase_region_health_canary_slow_run_alert_enabledhbase_canary_alert_unhealthy_region_percent_threshold For more information, see Enabling HBase region canary. This is available starting in COD v1.27 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

When you create an operational database, you can now define the structure of your database based on a predefined Data Lake template. The template defines the number of gateway, master, and worker nodes to be added while creating a database.You can select a template and accordingly the nodes are added into the COD cluster after the database is successfully created.To know more about this, see Creating a database using COD. This is available starting in COD v1.27 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD now allows you to configure JWT (JSON Web Token)-based authentication for your HBase clients, which uses an unique identifier and is a standard way of securely transmitting signed information between two parties. To know more about the JWT authentication, see Configuring JWT authentication for HBase client. This is available starting in COD v1.27 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

Apache Flink UpgradeDeployments can now upgrade from Flink 1.14 to 1.15.1. This update includes 40 bug fixes and a number of other enhancements. To learn more about what has been fixed check out the release notes. SQL Stream Builder UIThe Streaming SQL Console (UI) of SQL Stream Builder has been completely reworked with new design elements. The new design provides improved user access to artifacts that are commonly used or already created as part of a project simplifying navigation and saving the user time.   Software Development Lifecycle support with ProjectsProjects for SQL Stream Builder that improves upon the Software Developer Life Cycle needs of developers and analysts writing applications, allowing them to group together related artifacts and sync them to GitHub for versioning and CI/CD management. Before today, when users would create SQL jobs, functions, and other artifacts there was no effective way to migrate them to another environment for use (ie:

COD now allows you to create or update a database using a custom image. Custom images can be used for various purposes, such as compliance or security requirements. An image catalog is used to hold one or more custom images. You can inherit pre-installed packages or software from the custom image while creating or updating an operational database.You can also switch an image catalog of an existing operational database. For more information, see Managing custom images in COD. This is available starting in COD v1.25 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD now allows you to disable Kerberos authentication while creating an operational database using COD CLI. You can use the --disable-kerberos option while running the create-database command to disable the Kerberos authentication.cdp opdb create-database --environment-name ENVIRONMENT_NAME --database-name DATABASE_NAME --disable-kerberos This is available starting in COD v1.26 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD now allows you to create edge nodes while creating an operational database. You can define the number of edge nodes to be created, and COD automatically creates and configures the nodes in your COD cluster.You can also add additional nodes into your COD cluster or delete an existing one using the edge node instance ID.For more information, see Managing edge nodes. This is available starting in COD v1.26 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD introduces a CldrCopyTable utility, which is Cloudera’s version of the upstream CopyTable utility. This utility is an extension of the COD replication plugin that enables you to copy data cross-realm. This utility enables you to close any replication gaps that might arise depending on how you start or stop replication. This is available starting in COD v1.13 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

The CDP Public Cloud Release Summary for the month of December is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services.   See Public Cloud Release Summary - December 2022  

 We are pleased to announce the release of Cloudera ODBC Connector 2.6.17.1026 for Impala. This release addresses the following: Enhancements and New Features[IMP-927] Updated SSOIgnoreDriverNoPrompt default value[IMP-921][00395874] SAML SSO cookie cache support[IMP-857] Updated OpenSSL support Resolved Issues[IMP- 924][00394280] When the Delegation UID is set, the server parameters are not effected[IMP- 892][00356762] The connector returns successfully during SQLPrepare but throws an error, "Unsupported query" during SQLExecute for some queries Getting Started with the Cloudera DriverRead the ODBC Connector 2.6.17 for Apache Impala Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

Here are some highlights from the month of November 206 new support questions 6 new community articles 660 new members Rank  Community Article Author Components/ Labels #1 Datagen - Data Generator tool built for CDP Francois Risch  @frisch  Apache Hadoop Apache HBase Apache Hive Apache Impala Apache Kafka Apache Kudu Apache Solr Cloudera Data Platform (CDP) Cloudera Data Platform Private Cloud (CDP-Private) Cloudera Manager #2 Connecting CML to Data Lake & CDW with R Eduard Bangga @eduardb  Apache Hive Apache Impala Apache Spark Cloudera Data Science and Engineering Cloudera Data Science Workbench (CDSW) Cloudera Machine Learning (CML) #3 CDP Private Cloud 7.16 : HUE MYSQL 8 connection error Unexpected error. Unable to verify database connection Arshadullah Khan @khanarshadazmat  Cloudera Hue   We would like to recognize the below community members and employees for their efforts over the last

You can find the release notes and the download links in the documentation.Key features for this releaseRebase against NiFi 1.18 bringing the latest and greatest of Apache NiFi. It contains a ton of improvements and new features.Reset of the end of life policy: CFM 2.1.5 will be supported until August 2025 to match the CDP 7.1.7 LTS policy. It is particularly important as HDF and CFM 1.x are near to end of life.Parameter Providers: we are introducing the concept of Parameter Providers allowing users to fetch the values of parameters from external locations. In addition to a better separation of duties, it is also very useful to make CI/CD better and easier. With this release, we're supporting the following Parameter Providers:AWS Secret ManagerGCP Secret ManagerHashiCorp VaultDatabaseEnvironment VariablesExternal fileRegistry Client to connect to a DataFlow Catalog. The registry endpoint is now an extension in NiFi. It means that it is no longer limited to accessing a NiFi Registry ins

The latest release (2.3.0-b347) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces the following new features for both, AWS and Azure customers: Flow Designer [Technical Preview]Developers can now build new data flows from scratch using the integrated Designer. Flow Drafts can be tested using Test Sessions before they are published to the Catalog. Once they are in the catalog, users can create Flow Deployments using the Deployment Wizard or CLI.Note:The Flow Designer is a Technical Preview feature and requires an entitlement to be activated. Reach out to your Cloudera Account team to get Technical Preview access. Added support for certificate renewal of inbound connection endpointsSix new ReadyFlows have been added to the ReadyFlow Gallery:Airtable to S3/ADLSBox to S3/ADLSDropbox to S3/ADLSGoogle Drive to S3/ADLSMQTT to KafkaSalesforce to S3/ADLSDataFlow is now also available in the EU and APAC regional CDP control planes.Updated to support using kubernetes 1.23 EKS a

The ML Runtimes 2022.11-1 Release includes the GA version of the new workbench architecture, the PBJ (Powered by Jupyter) Workbench. In the previous Workbench editor, the code and output shown in the console (the right-hand pane in the image below) were passed to and from Python, R, or Scala via a Cloudera-specific, custom messaging protocol. In the PBJ Workbench, on the other hand, the code and output are now passed to and from the target language runtime via the Jupyter messaging protocol. They are handled inside the runtime container by a Jupyter kernel and rendered in your browser by JupyterLab’s client code. This may seem like a subtle change, but it will provide CML users with some major benefits. First, the behavior of user code and third-party libraries on CML will be more consistent with its behavior in Jupyter-based environments. That means that a wider variety of rich visualization libraries will work out of the box, and in cases where rich visualiza

The CDP Public Cloud Release Summary for the month of November is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services.  See CDP Public Cloud Release Summary - November 2022

We're pleased to announce the release of Cloudera Private Cloud (PVC) Data Services 1.4.1, which delivers key new features and improvements across the platform for better enterprise readiness and BI/AI outcomes.  This release appeals to both data practitioners and platform administrators with the following key highlights: Significant new functionality in Data Services such as Quota Management (CDE), support for Spark History Server (CDE), use of deterministic namespace in Kerberos principals (CDW), ability to use CDP CLI to install and manage CDW Private Cloud (CDW). Promoting enterprise readiness with new capabilities such as External Docker Registry, rolling restart of ECS nodes during upgrades, fixes for Critical CVEs, and additional platform certifications.  Please see the Release Notes for CDP PVC Data Services 1.4.1 for the full list of features delivered in this release.  Cloudera’s Professional Services team is available to ensure yo

CML Experiments have been rebuilt, leveraging the MLflow ecosystem to complement CML’s existing strengths in model development and deployment. CML now ships the mlflow SDK and an integrated visual experience that enables experiment tracking and comparison via flexible visuals.  Model development is an iterative process that involves many “experiments” to determine the right combination of data, algorithm, and hyperparameters to maximize accuracy. Multiple versions of the “model” are created as the iterative process continues after the initial deployment. To efficiently develop models while addressing the technical and governance needs for traceability and repeatability, data scientists need to be able to track these different experiments and model versions and see which experiments produced which models. CML Experiment Tracking through MLflow API CML’s experiment tracking features allow you to use the MLflow client library for logging parameters, code versions, metrics,

Here are some highlights from the month of October 217 new support questions 15 new community articles 702 new members   Rank  Community Article Author Components/ Labels #1 Using Flink to monitor Ranger audits ‚Äì User Session Analysis William Dyson @wdyson Apache Flink Apache Ranger Cloudera Streaming Analytics (CSA) #2 Dynamic Policy Hooks in CDP Davide Vergari @dvergar  Apache Ranger #3 Understanding NiFi's "Terminate" option on running components. Matthew Clarke @MattWho  Apache NiFi #4 Parquet timestamp handling for non UTC data in CDP 7.1.7 cluster. Usman Khan @usman35  Apache Hive  Apache Spark Cloudera Data Platform (CDP) Cloudera Data Platform Private Cloud (CDP-Private) Cloudera Enterprise Data Hub #5 Override CDSW image load directory other than /var/lib/cdsw/docker-tmp Gopinath Nagarajan @Gopinath  Cloudera Data Science Workbench (CDSW)

The CDP Public Cloud Release Summary for the month of October is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services.  See CDP Public Cloud Release Summary - October 2022.