With the cdpctl command line utility, Cloud Administrators can verify compatibility of existing setups with CDP prior to handing over to the CDP Administrator.  cdpctl now adds support for Microsoft Azure in addition to Amazon Web Services.     CDP Public Cloud customers often choose to provision their environments into existing cloud accounts where resources such as networking, IAM and storage are pre-created by IT and vetted for adherence to corporate standards.   Getting this right often takes several iterations between the Cloud Administrator and the CDP Administrator. In larger organizations with well-defined organizational boundaries, this exercise can stretch into days.   Cloud Administrators can now download the cdpctl utility and use it to verify compatibility of existing resources with CDP on both Azure and AWS.  The leading cause of CDP provisioning failures are related to network, storage locations or access to these (IAM).  This release of cdpctl validates cloud infrastructure against a list of our most common failure scenarios, including verifying networking (VPC/VNet, Subnets, Security Groups), storage (S3/ADLSv2) locations and validating that the provided cloud IAM resources have the right access to the underlying resources.    Future releases will add the ability to create missing resources and provisioning CDP environments,.   To use cdpctl, you must have Docker installed on your computer.  Instructions on how to get started are available in the preview section of the Cloudera Docs site, or visit the cdpctl repository on the Cloudera Labs github https://github.com/cloudera-labs/cdpctl     ... View more

With the cdpctl command line utility, Cloud Administrators can verify compatibility of existing setups with CDP prior to handing over to the CDP Administrator.  This will greatly reduce the probability of errors during onboarding into an existing cloud account.     CDP Public Cloud customers often choose to provision their environments into existing cloud accounts where resources such as networking, IAM and storage are pre-created by IT and vetted for adherence to corporate standards.   Getting this right often takes several iterations between the Cloud Administrator and the CDP Administrator. In larger organizations with well-defined organizational boundaries, this exercise can stretch into days.   Cloud Administrators can now download the cdpctl utility and use it to verify compatibility of existing resources with CDP.  This release of cdpctl validates AWS infrastructure against a list of our most common failure scenarios, including verifying networking (VPC, Subnets, Security Groups), simulating IAM policies for validity and storage (S3 bucket locations).  Future releases will add the ability to create missing resources, provisioning CDP environments, and extend support to other clouds (Azure, Google Cloud).   To use cdpctl, you must have Docker installed on your computer along with a configured awscli.  cdpctl will access credentials stored in your ~/.aws directory to run the validations.   To get started, please visit the cdpctl repository on the Cloudera Labs github https://github.com/cloudera-labs/cdpctl ... View more

The Management Console for CDP public cloud improves support for hybrid environments by adding support for CDP Private Cloud Base.    You can now register Private Cloud Base clusters as Classic Cluster, similar to what has been available for CDH and HDP clusters. You do not need inbound connectivity to a cluster running on-premise; the process supports creation of a reverse ssh tunnel if required.    This capability will allow you to use Private Cloud Base as the source for Replication Manager.     This feature has been enabled for all CDP accounts.  To learn how to use this feature, please view the documentation.   Support for Data Catalog is also available as a Preview, but Data Catalog and Replication Manager support are mutually exclusive while this feature is in Preview state. Preview features are available by request only. To get this feature enabled in your CDP account, please reach out to your Cloudera sales team.  ... View more

The Management Console already provides the capability to generate CLI commands for credential creation (AWS, Azure) as well as registering a new SDX/Data Lake (AWS, Azure).  This greatly simplifies the process of automating environment creation.  With this new feature, this functionality is now extended to Data Hub and makes the process of automating Data Hub cluster creation.     You can either get the command to create a replica of an existing Data Hub cluster, or from a  Data Hub Cluster Definition.     To setup the cdpcli, follow the instructions in our documentation.  A full set of cdpcli instructions is available here.   ... View more

The default root volume size for all instances on all cloud providers has been increased to 100GB.  Previously, CDP used different volume sizes depending on the cloud service provider.     You can override this by choosing custom hardware and storage for your data hub clusters by following the instructions in our documentation for Amazon Web Services , Microsoft Azure and Google Cloud.     ... View more

A number of new features (Endpoint Access Gateway, Medium Duty SDX) have resulted in CDP exercising a set of AWS APIs that were not used earlier. The default cross account role (available via the CDP Documentation) already includes these APIs and no action is required.  However, customers who are running a custom cross account role policy may need to update their policy to ensure they have added the following actions.  Failure to do so will result in environment creation operations failing.   cloudformation:UpdateStack cloudformation:ListStackResources elasticloadbalancing:DescribeLoadBalancers elasticloadbalancing:DescribeTargetHealth elasticloadbalancing:RegisterTargets elasticloadbalancing:DeregisterTargets   For details on how to find the AWS Cross Account role for your environment, please see the documentation on how to change an environment's credential.  For details on finding the Amazon Resource Name of the AWS IAM Role, please see the documentation on modifying a provisioning credential.   ... View more

Management Console now allows you to select specific nodes within a node group to include a repair operation.. This will reduce the downtime incurred if only a subset of the nodes are unhealthy.     The CDP Management Console has always provided the ability to repair nodes, but this has been at a host group level. With the recent addition of Medium Duty SDX clusters, this would result in the repair operation happening on both healthy and unhealthy nodes, resulting in a service outage while the repair operation was in progress.     From the Hardware tab of the Data Lake details, you can click the Repair icon to select specific nodes within a host group to repair.   For details on what happens during the repair process, please see the documentation on Data Lake Repair. ... View more

You can now run Data Hub clusters in your Google Cloud Platform account.  To make it easier to get started with GCP on Google Cloud, a Quick Start is available that will walk you through the prerequisite set up required in your cloud provider account, as well as creating a CDP credential and registering an environment in CDP. This will greatly reduce the amount of time it takes to set up a Proof of Concept for CDP.    CDP requires a number of pre-requisites to be set up in your GCP Project before an environment can be created.  The Quick Start includes GCP Deployment Manager templates to automate creation of pre-requisite resources including VPC network, a subnet, firewall rules, service accounts and storage buckets.     You can get started by reading the Quick Start on Google Cloud documentation.  Alternatively, if you'd prefer to set things up yourself, you can follow the documentation on working with a Google Cloud environment.    Similar Quick Start guides are also available for AWS and Azure. For more details, please visit the Quick Starts section of the documentation.  ... View more