Announcements

Here are some highlights from the month of September 257 new support questions 7 new community articles 645 new members Rank  Community Article Author Components/ Labels #1 Real-time Twitter Dashboard using Cloudera Data Platform Anshul Gupta  (@Anshul_Gupta) Apache Hive Apache NiFi Cloudera Data Platform (CDP) Cloudera DataFlow (CDF) #2 Connecting DBeaver to a CDP Impala Virtual Warehouse Craig Nelson  (@cnelson2) Apache Impala Cloudera Data Platform (CDP) #3 Accelerate COVID-19 outreach programs using Cloudera Data Platform - Public Cloud Anshul Gupta (@Anshul_Gupta) Apache Atlas Apache Hive Apache NiFi Cloudera Machine Learning (CML) #4 Using Streams Messaging Manager in CDP to create a Snowflake KConnect Sink Craig Nelson ( (@cnelson2) Apache Kafka Cloudera Data Platform (CDP) Kerberos #5 Configuring Kafka Console Producer/Consumer on a Secured CDP Cluster Craig Nelson (@cnelson2) Apache Kafka Cloudera

COD now supports the CDP CLI commands update-hbase-configuration and describe-hbase-configuration to update and retrieve the HBase configuration values.For more information, see update-hbase-configuration and describe-hbase-configuration. This is available starting in COD v1.24 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

For Multi-AZ deployments, COD assigns the nodes to multiple regions when multiple strong meta servers exist in the deployment. This is available starting in COD v1.24 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

Cloudera Manager and monitoring systems require more resources than a regular deployment for big clusters with a large number of nodes. To support this, now COD supports a fast gp2 storage when you choose a gateway node of HEAVY type. This is available starting in COD v1.24 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

The CDP Public Cloud Release Summary for the month of September is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. See CDP Public Cloud Release Summary - September 2022. 

As businesses continue to adopt and build open lakehouses built with Apache Iceberg on CDP, data scientists need easy access to these new table formats, so they don’t spend their time figuring out connection dependencies and configurations. Cloudera Cloudera AI’s Data Connection and Snippet support simplify data access in CDP. Data scientists can use the cml.data library to gain access to a Data Lake via Spark or query their Virtual Warehouse with Hive or Impala. With recent improvements to the cml.data library, CML Snippets now fully support the Iceberg table format for all Spark, Hive, and Impala data connections. To learn more about Data Connection and Snippet read the following article:https://blog.cloudera.com/one-line-away-from-your-data/

Cloudera Cloudera AI now provides a built-in dashboard for monitoring technical metrics relating to deployed CML Models, such as request throughput, latency, and resource consumption. When machine learning models are deployed in production, it’s essential to know whether the model is successfully providing responses to all queries within the required timeframe and to be able to investigate and find the root cause for any failed responses or other downstream issues. Further, it can be challenging to know ahead of time the resource requirements for the model, such as the number of replicas, and amount of memory and CPU allocated to each replica. This can make it difficult to find the balance between the risk of underprovisioning resources leading to slow responses or timed out requests, and the risk of unnecessarily wasting resources that could be used for other workloads.   The new monitoring features for CML Models provide observability that makes these challenges m

Here are some highlights from the month of August 247 new support questions 3 new community articles 639 new members   Rank  Community Article Author Components/ Labels #1 How Cloudera Data Platform excels at hybrid use cases Anshul Gupta @Anshul_Gupta  Apache Hive Apache NiFi Apache Spark Cloudera Data Platform (CDP) Cloudera Data Platform Private Cloud (CDP-Private) #2 Using Recipes to Simplify Data Hub Setup Tasks Craig Nelson  @cnelson2  Cloudera Data Platform (CDP) Cloudera Enterprise Data Hub #3 Knox HA / Loadbalancing using Haproxy Vikas Srivastava  @cloude  Apache Knox Hortonworks Data Platform (HDP)   Congratulations Robin Kakkasery @rki_  for being the Community Spotlight member for the month!! Thanks for everything that you do!!Check out the Community Member Spotlight on Cloudera Linkedin!     We would like to recognize the below community members and employees f

 We are pleased to announce the release of Cloudera JDBC 2.6.21 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[00371927][HIVEJ-733] Updated SAML SSO authentication behavior Resolved Issues[00391275][HIVEJ-753] After closing HTTP connections, the connector does not clean up native threads Getting Started with the Cloudera DriverRead the Cloudera Hive JDBC 2.6.21 Driver for  Apache Hive Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

 We are pleased to announce the release of Cloudera ODBC 2.6.13 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[HAR-1225] [HAR-1227][HAR-1228] Updated platform support[00371927] [HAR-1288] New mechanism for SAML SSO[00374764] [HAR-1289] Override SQL_ATTR_QUERY_TIMEOUT[HAR-1294] Updated OpenSSL library[HAR-1302] Heartbeat messages[HAR-1303] Configure query translation for CTAS syntax[HAR-1305] Configuring the DSN file Resolved Issues[HAR-1220] When using a new TCP connection to make a HTTP request, the connector does not honor cookies[HAR-1287] The HTTP connection client does not reinitialize after an SSL_read error[HAR-1300] In some cases, when using Linux, the connector terminates unexpectedly when resolving host names[HAR-1301] When using UnixODBC on Linux, error messages are truncated[HAR-1304] The connector does not allow the use of server and intermediate certificates that do not have a CRL distributio

CDP SCIM Support for Azure Active DirectoryThe identity team at Cloudera has been working to add the System for Cross-domain Identity Management (SCIM) support to Cloudera Data Platform (CDP) and we're happy to announce the general availability of SCIM on Azure Active Directory! If you know what SCIM is and you want to start using it then here's a link to our docs to get started. Otherwise, read on. It might just save you a bunch of time and hassle. This blog is broken into two parts:Part One (this blog): CDP SCIM Support for Active Directory, which discusses the core elements of CDP’s SCIM support for Azure AD and the pain points that SCIM solves, and Part Two (now up) : SCIM: An Introduction, for readers who might not be familiar with the SCIM standard.  SCIM support on CDP helps ease and, in many cases, resolve the pain points of our customers. Some of these pain points are listed below.  Pain point: user management as code The main win is that we'

We are pleased to announce the release of Cloudera JDBC Connector 2.6.29 for Impala.  This release has the following enhancements and fixes: Enhancements and New Features[00371927][IMPJ-697] Updated SAML SSO authentication behaviour Resolved Issues[00391275][IMPJ-713] After closing HTTP connections, the connector does not clean up native threads Getting Started with the Cloudera DriverRead the Cloudera JDBC Connector 2.6.29 for Impala Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

The latest release (2.2.0-b194) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces the following new features for both, AWS and Azure customers: A streamlined first time user experienceYou can use the new Hello World ReadyFlow to create your first flow deployment without dependencies on source or target systems. New landing pages for the Dashboard and Catalog allow you to get started with just one click.Four additional new ReadyFlows are available in the ReadyFlow GalleryNon-CDP ADLS to S3/ADLSNon-CDP S3 to S3/ADLSListenTCP filter to S3/ADLSListenSyslog filter to S3/ADLSEndpoint Hostnames certificates can now be renewed when using Inbound ConnectionsLearn more about the new features in the CDF-PC documentation and take an interactive tour of CDF-PC.

COD now displays notifications about all the major events on the COD UI. For example, auto-scaling events, auto-healing events, and alerting notifications.The UI notifications is a major step in providing enough information on COD UI so that you do not need to navigate to multiple internal systems used by COD to get the required information as well as be aware of the events happening in the system on an immediate basis. This is available starting in COD v1.23 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD allows you to customize the attached storage worker node for an HDFS cluster while creating a database. You can define the EBS volumes for HDFS clusters.You can also view the customized attached storage worker nodes using the ‘describe-database’ and ‘list-database’ commands. This is available starting in COD v1.23 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

We're pleased to announce the release of Cloudera Private Cloud (PvC) Base 7.1.8 and Cloudera Manager 7.7.1, which introduce key new features to improve the analytics capabilities offered to your business users, enhanced enterprise readiness, as well as additional third party support. This is a cumulative maintenance release that carries forward the features from 7.1.7 Service Pack 1 (SP1) and prior releases.  Here are some highlights of 7.1.8 we’d like to draw your attention to:This release has a concerted focus on platform resilience, delivering vastly improved high availability (HA) of the platform, which rings true to our laser focus on enterprise readiness.  As organizations plan for continuity in the face of an unplanned outage, 7.1.8 promotes HA with Cloudera Manager's support for active-passive HA, replication between Ozone clusters, balanced distribution of data with Ozone, standby NameNodes for added fault-tolerance, Hive ACID table replication for disaster rec

Here are some highlights from the month of July 231 new support questions 8 new community articles 586 new members Rank  Community Article Author Components/ Labels #1 MySQL CDC with Kafka Connect/Debezium in CDP Public Cloud @cnelson2  Apache Kafka Cloudera Data Platform (CDP) Kerberos #2 Event driven pipelines in Azure with CDE(Cloudera Data Engineering) orchestrated by Azure Event Grid/Azure Functions @hrongali  Apache Spark Cloudera Data Engineering (CDE) Cloudera Data Platform (CDP) #3 How to configure CML's Spark Connection @peter_ableda  Cloudera Data Science Workbench (CDSW) Cloudera Machine Learning (CML) #4 HDP to CDP - Atlas backup and restore @hpasumarthi  Apache Atlas Cloudera Data Platform (CDP) Hortonworks Data Platform (HDP) #5 "kinit: Preauthentication failed while getting initial credentials" when using Active Directory or FreeIPA @araujo  Kerberos Securit

The CDP Public Cloud Release Summary for the month of July is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. See CDP Public Cloud Release Summary - July 2022. 

Over the last few quarters, more and more of our customers deployed their production workloads on Cloudera Cloudera AI. Some of them rely on CML with their predictive maintenance use case, others predict churn or detect fraudulent transactions. The common thing between them is that the ml workloads running on CML are critical for their business’s success.   The new CML Backup and Restore capability gives you an extra layer of protection by giving you the ability to resume operations in a timely manner following an outage or crisis. Now, administrators can take on-demand backups of CML workspaces before cluster operations like an upgrade and do periodic backups during off-peak hours. Backed-up CML workspaces can be restored into a new CML workspace in the same or a new CDP environment, and all project artifacts like deployed models and applications will be recovered.  The Backup and Restore capabilities are available on AWS today, and we are planning to roll out

Key features for this releaseAirflow stability and performance. With this release we now use the latest stable Airflow release 2.2.5.  In conjunction with service enhancements, our testing indicates improved stability under higher DAG and task loads. This allows DAG loads to reach up to 1000 per VC, combined with a concurrency of 200-300 parallel Airflow tasks within the same Service. Pre-loaded sample jobs and data for new users To help new users self-learn CDE, all new Virtual Clusters will have the option to load example Airflow & Spark jobs accompanied with sample data.Spark 3 support for raw scala code Previously this feature was limited to Spark 2, it is now extended to Spark 3 based Virtual Clusters.   This allows users to directly run raw scala via API & CLI in batch-mode without having to compile, similar to what spark-shell supportsPipeline UI editor for Airflow is now GA with support for all major browsers (Firefox, Chrome, and Safari). New Virtua

Data Scientists have access to a wide range of ML Runtimes, they can use different versions of Python, R, and Scala kernels with the Workbench or the JupyterLab editors and can benefit from extended capabilities like GPU acceleration. Cloudera continues to release new versions of ML Runtimes, while customers can also register custom ones they build to solve their specific business use case. With the continuous additions, the available options can grow large, and some will become irrelevant or outdated.  CML now supports administrators to disable Runtime variants or specific versions. For example, they can decide to disable all of the Python 3.6 Runtimes as the python kernel is officially EOL and won't receive any further security and bugfix patches. Usage of these runtimes can be considered a security risk that administrators now can solve. Once administrators disable an ML Runtime, data scientists won't be able to use them for development, and existing workloads configured w

Here are some highlights from the month of June   206 new support questions 3 new community articles 646 new members     Rank  Community Article Author Components/ Labels #1 NiFi cluster sandbox on Docker @araujo  Apache NiFi #2 Decompressing nested ZIP files in NiFi @araujo  Apache NiFi #3 Cloudera Machine Learning adds ability for admins to add a static subdomain for the CML Workspace @cpv0310  Cloudera Machine Learning (CML) #4 MySQL CDC with Kafka Connect/Debezium in CDP Public Cloud  @cnelson2  Apache Kafka Cloudera Data Platform (CDP) Kerberos   Thank you to @shehbazk for supporting our community! :trophy: 👏 Check out the Community Member Spotlight on Cloudera Linkedin!     We would like to recognize the below community members and employees for their efforts over the last month to provide community solutions.    See all our top participants

The CDP Public Cloud Release Summary for the month of June is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. See CDP Public Cloud Release Summary - June 2022. 

The latest release (2.1.0-b123) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces the following new features for both, AWS and Azure customers: [Technical Preview] CDF service and deployment upgrades are now supported.CDF service upgrades require a specific entitlement that is granted by request only. Reach out to your Cloudera Account Representative or open a support ticket to request CDF service upgrades.The create-deployment CLI command now supports Inbound Connections.CDF now correctly configures the deployment when it detects any of the two reserved Controller Services Default NiFi SSL Context Service and Inbound SSL Context Service regardless of whether they are external references or not.Kubernetes Version 1.22 is now the default Kubernetes version for CDF. When CDF is enabled, it creates AWS EKS and Azure AKS clusters based on version 1.22.CDF now supports Data Lake scaling. After the Data Lake scale operation has completed, affected

Certain countries may have regulatory requirements that limit or forbid specific types of data from being sent or stored outside of the country or region where an organization operates. For this reason Cloudera has introduced new regions in which the Control Plane can run. By choosing a region other than us-west-1, you can ensure that sensitive metadata (such as information about CDP users) does not leave its region or country of origin. For more information, see CDP Control Plane regions. This is available starting in COD v1.22 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).