Cloudera for Azure supports CMK with Azure Key Vault. This article describes how to set up a network for CDP for Azure to allow a successful Data Service deployment. There are two options to deploy Azure Key Vault access endpoints, public and private. With public deployment, Azure Key Vault FQDN is resolved into a public IP address. With private deployment, a private endpoint for Azure Key Vault must be created as well as a private DNS zone for Azure Key Vault.  Some Azure users have governance requirements to mandate a private endpoint for Azure Key Vault. CDP environment and DataHub clusters support Azure Key Vault private endpoints. However, Cloudera Data Services for Azure do not support private endpoints.  Best practice To support both Cloudera Data Services and Data Hubs, please make sure to configure the networking  configuration for the Azure Key Vault to be like the picture below. Do not choose the second and the private endpoint option. The reason will be discussed in the following section.   For all the subnets to be used by CDP, make sure to configure a service endpoint for Azure Key Vault, so that the traffic will be routed to the Key Vault through the Azure backbone instead of organization's firewall.  Reason for not being able to use private endpoints or Azure KeyVault Firewall Cloudera Data Services use Azure K8s as compute infrastructure. AKS is a managed service. It deploys two parts of components in Azure subscriptions.  The compute resources are in VMSS to be deployed in the customer’s subscription. The ETCD is to be deployed in a backend Azure managed subscription. The ETCD has a public IP address. Users and workloads can access the ETCD through a private endpoint. From the ETCD point of view, it is inbound traffic. However, the egress traffic always goes through the public IP address.  Azure provides 3 alternatives for the ETCD service to access Azure Key Vault.    Option 1:  Add Key Management Service (KMS) etcd encryption to an Azure Kubernetes Service (AKS) cluster (legacy) Legacy, to be deprecated in the future. In this option, ETCD accesses Azure Key Vault through the public internet. Because the ETCD public IP cannot be obtained, users cannot enable Key Vault firewall to prevent access from other sources. This is what Cloudera is using right now. Option 2:  Enable KMS data encryption in Azure Kubernetes Service (AKS) clusters (Preview) There is a service tag associated with the public IP address that is being used by ETCD. Users can enable the Key Vault Firewall to allow the service tag.  However, this feature is still in preview stage and Cloudera cannot invest development effort to support it.  Option 3: Use API Server VNet Integration in Azure Kubernetes Service (AKS) Azure launched AKS VNET integration which will be the best solution for this requirement. In this solution, Azure assigns a private IP address to the ETCD service, which allows the ETCD to access the Key Vault through either service endpoint or private endpoint.  However, this feature is not yet available in all the regions.  ... View more

Requirement introduction Azure PostgresDB is a PaaS hosted on Azure. Azure hosts PaaS services in an Azure backend subscription. The resources created in the customer’s subscription are actually descriptions and metadata of the actual compute resources in the Azure backend subscription.  When the Azure backend subscription reaches its capacity limit, Azure will create new resources in a new Azure backend subscription. However, existing PaaS resources will remain in the old Azure backend subscription. When a CDP  user stops a CDP environment, the compute resources for the PostgresDB will be deallocated from the backend Azure subscription, and have to compete for compute resources when being restarted. Suppose it cannot get resources from the backend Azure subscription, as it is already out of capacity. In that case, it cannot be started, and consequently blocks the environment from being started.  The resolution provided by Microsoft is to back up and restore the database. The restored Postgres DB will allocate the compute resource from a new Azure backend subscription.  To keep the CDP environment working, the restored DB must have the same database name as the original database.  High-Level Steps: Stop the environment. Take an ondemand backup of each Azure PG DB and restore the DB to a new PG DB name. (e.g., restored-db) Take ondemand backup of newly created DBs, restored-db  Delete the original PG DBs Restore the PG DBs to the original names from the new backup of the restored-db. Adjust for HA. (Note the original HA configuration, as they may not be the same across) Start the environment. Detailed Steps: 1. Backup Existing Server  From Cloudera Management Console → Environments (select yours) → Data Lake tab → Database (copy value) Save this value to a notepad on your computer. Repeat the process for Datahub Postgres DB instances and COD instances. 2. Turn off the cluster (and associated datahubs). Do not proceed until it’s off. Locate the same Postgres DB in the Azure Portal, and start the database. With the server ON, navigate to Settings >> Backup and restore and select Backup. (gets the latest backup). Initiate an ondemand FULL backup. Note: None of the existing backups can be used. With the server ON, navigate to Settings >> Backup and restore and select Backup. (gets the latest backup). Initiate an ondemand FULL backup. Note: None of the existing backups can be used. Verify completion (will take several mins): IMPORTANT! Verify HA config (Check for Zone Redundant or Same Zone) 3. Perform DB “Restore” to create Temporary Database Select the Restore icon on the Datalake DB. Select Fast restore point (Restore using full backup only) – Use the latest one we just took! Add restored-db to the server name syntax. (Each restore name should be identifiable to map back to the dbsvr-... original name. If the DB name is already taken, change it to a different name. Then make sure networking aligns with the existing PostgreS DB. Verify and create. Verify completion Repeat step 2 process for Datahub Postgres DB instances. Backup NEW Server to create an Ondemand Backup Create ondemand backup for this new database: Repeat step 3 process for Datahub Postgres DB instances. ONLY AFTER COMPLETION OF THIS STEP MOVE FORWARD (Verify successful backup!) 4. Delete OLD DB Server Repeat step 4 process for Datahub Postgres DB instances. 5. Restore the NEW server to the OLD server name. Important! Make sure to use the same server name the original database had. See example: Make sure the original network settings copy over: Verify completion: Repeat step 5 process for Datahub Postgres DB instances. 6. High Availability Between backups/restores, HA is turned off, so check for High Availability turned on. Certain Postgres DBs get assigned Zone Redundant, and others are assigned Same Zone so pay attention to original config.  Verify deployment completion:   Repeat step 6 process for Datahub Postgres DB instances. 7. Restart the CDP environment (and clean up) Restart the CDP environment and run basic sanity checks on Cloudera Manager services. After verifying successful completion, delete the temporary Postgres DB. Repeat this step for any other temporary datahub Postgres DB as well.   ... View more

Abstract: CDP is using many managed cloud services from different CSPs as underlying infrastructure. The DNS configuration in a typical CDP onboarding is critical to a successful deployment. This article introduces some common DNS setup issues with AWS and corresponding solutions. AWS services with private endpoints used by CDP  Most enterprise AWS consumers are using HUB-SPOKE network architecture with custom private DNS. The Custom private DNS usually resides on HUB VPC. Correct DNS configuration needs to be created for a successful deployment. AWS managed services CDP is using RDS Elastic Kubernetes Service Elastic File Service In a typical HUB-Spoke network, the custom private DNS server is usually deployed in the HUB network. There are several other DNS servers that are being used.  AWS Public DNS: Hosted by AWS on internet AWS Default Private DNS: One per VPC. At 169.254.169.253  Different AWS services use different DNS for its domain registration.     AWS RDS create an endpoint in the CDP VPC, and the DNS record for this endpoint is registered in AWS Public DNS AWS EKS register an endpoint in the CDP VPC, and the DNS record for this endpoint is registered in AWS Public DNS AWS EFS create an endpoint in the CDP VPC, and the DNS record for this endpoint is registered in AWS VPC Default DNS Challenges introduced to CDP deployment and resolution When using AWS VPC default DNS for CDP VPC The VPC default DNS can return all the resolves for RDS, EKS, and EFS endpoints. When using custom private DNS in the same VPC as CDP resources Recursive lookup for RDS and EKS is DNS native feature, and custom private DNS will recursively lookup the DNS record and come back from AWS public DNS. But because the EFS DNS record is in VPC Default DNS, it can’t be recursively looked up. Users will need to manually configure conditional forward from the custom private DNS to the EFS endpoint.   Recursive lookup for RDS and EKS is DNS native feature, and custom private DNS will recursively lookup the DNS record and come back from AWS public DNS. But because the EFS DNS record is in VPC Default DNS, it can’t be recursively looked up. Custom DNS on another network cannot do conditional forward to the AWS VPC default DNS for CDP VPC. Users will have to create an AWS Route 53 Resolver with an inbound resolve endpoint, so that the custom private DNS can forward the conditional forward for EFS to this Resolver inbound endpoint.   So far, Data Warehouse doesn’t support custom private DNS. Which means if the customer’s network team does not allow VPC Default DNS, Data warehouse cannot be deployed. Sometimes, we can convince the network team to use AWS VPC default DNS, but that will introduce another dilemma that the resources in CDP VPC will not be able to resolve the other resources in the custom private domain. Users can create an AWS Route 53 resolver with outbound endpoint and forward the requests to the custom private domain to the custom private DNS server.     ... View more