Requirement introduction Azure PostgresDB is a PaaS hosted on Azure. Azure hosts PaaS services in an Azure backend subscription. The resources created in the customer’s subscription are actually descriptions and metadata of the actual compute resources in the Azure backend subscription.  When the Azure backend subscription reaches its capacity limit, Azure will create new resources in a new Azure backend subscription. However, existing PaaS resources will remain in the old Azure backend subscription. When a CDP  user stops a CDP environment, the compute resources for the PostgresDB will be deallocated from the backend Azure subscription, and have to compete for compute resources when being restarted. Suppose it cannot get resources from the backend Azure subscription, as it is already out of capacity. In that case, it cannot be started, and consequently blocks the environment from being started.  The resolution provided by Microsoft is to back up and restore the database. The restored Postgres DB will allocate the compute resource from a new Azure backend subscription.  To keep the CDP environment working, the restored DB must have the same database name as the original database.  High-Level Steps: Stop the environment. Take an ondemand backup of each Azure PG DB and restore the DB to a new PG DB name. (e.g., restored-db) Take ondemand backup of newly created DBs, restored-db  Delete the original PG DBs Restore the PG DBs to the original names from the new backup of the restored-db. Adjust for HA. (Note the original HA configuration, as they may not be the same across) Start the environment. Detailed Steps: 1. Backup Existing Server  From Cloudera Management Console → Environments (select yours) → Data Lake tab → Database (copy value) Save this value to a notepad on your computer. Repeat the process for Datahub Postgres DB instances and COD instances. 2. Turn off the cluster (and associated datahubs). Do not proceed until it’s off. Locate the same Postgres DB in the Azure Portal, and start the database. With the server ON, navigate to Settings >> Backup and restore and select Backup. (gets the latest backup). Initiate an ondemand FULL backup. Note: None of the existing backups can be used. With the server ON, navigate to Settings >> Backup and restore and select Backup. (gets the latest backup). Initiate an ondemand FULL backup. Note: None of the existing backups can be used. Verify completion (will take several mins): IMPORTANT! Verify HA config (Check for Zone Redundant or Same Zone) 3. Perform DB “Restore” to create Temporary Database Select the Restore icon on the Datalake DB. Select Fast restore point (Restore using full backup only) – Use the latest one we just took! Add restored-db to the server name syntax. (Each restore name should be identifiable to map back to the dbsvr-... original name. If the DB name is already taken, change it to a different name. Then make sure networking aligns with the existing PostgreS DB. Verify and create. Verify completion Repeat step 2 process for Datahub Postgres DB instances. Backup NEW Server to create an Ondemand Backup Create ondemand backup for this new database: Repeat step 3 process for Datahub Postgres DB instances. ONLY AFTER COMPLETION OF THIS STEP MOVE FORWARD (Verify successful backup!) 4. Delete OLD DB Server Repeat step 4 process for Datahub Postgres DB instances. 5. Restore the NEW server to the OLD server name. Important! Make sure to use the same server name the original database had. See example: Make sure the original network settings copy over: Verify completion: Repeat step 5 process for Datahub Postgres DB instances. 6. High Availability Between backups/restores, HA is turned off, so check for High Availability turned on. Certain Postgres DBs get assigned Zone Redundant, and others are assigned Same Zone so pay attention to original config.  Verify deployment completion:   Repeat step 6 process for Datahub Postgres DB instances. 7. Restart the CDP environment (and clean up) Restart the CDP environment and run basic sanity checks on Cloudera Manager services. After verifying successful completion, delete the temporary Postgres DB. Repeat this step for any other temporary datahub Postgres DB as well.   ... View more

Abstract: CDP is using many managed cloud services from different CSPs as underlying infrastructure. The DNS configuration in a typical CDP onboarding is critical to a successful deployment. This article introduces some common DNS setup issues with AWS and corresponding solutions. AWS services with private endpoints used by CDP  Most enterprise AWS consumers are using HUB-SPOKE network architecture with custom private DNS. The Custom private DNS usually resides on HUB VPC. Correct DNS configuration needs to be created for a successful deployment. AWS managed services CDP is using RDS Elastic Kubernetes Service Elastic File Service In a typical HUB-Spoke network, the custom private DNS server is usually deployed in the HUB network. There are several other DNS servers that are being used.  AWS Public DNS: Hosted by AWS on internet AWS Default Private DNS: One per VPC. At 169.254.169.253  Different AWS services use different DNS for its domain registration.     AWS RDS create an endpoint in the CDP VPC, and the DNS record for this endpoint is registered in AWS Public DNS AWS EKS register an endpoint in the CDP VPC, and the DNS record for this endpoint is registered in AWS Public DNS AWS EFS create an endpoint in the CDP VPC, and the DNS record for this endpoint is registered in AWS VPC Default DNS Challenges introduced to CDP deployment and resolution When using AWS VPC default DNS for CDP VPC The VPC default DNS can return all the resolves for RDS, EKS, and EFS endpoints. When using custom private DNS in the same VPC as CDP resources Recursive lookup for RDS and EKS is DNS native feature, and custom private DNS will recursively lookup the DNS record and come back from AWS public DNS. But because the EFS DNS record is in VPC Default DNS, it can’t be recursively looked up. Users will need to manually configure conditional forward from the custom private DNS to the EFS endpoint.   Recursive lookup for RDS and EKS is DNS native feature, and custom private DNS will recursively lookup the DNS record and come back from AWS public DNS. But because the EFS DNS record is in VPC Default DNS, it can’t be recursively looked up. Custom DNS on another network cannot do conditional forward to the AWS VPC default DNS for CDP VPC. Users will have to create an AWS Route 53 Resolver with an inbound resolve endpoint, so that the custom private DNS can forward the conditional forward for EFS to this Resolver inbound endpoint.   So far, Data Warehouse doesn’t support custom private DNS. Which means if the customer’s network team does not allow VPC Default DNS, Data warehouse cannot be deployed. Sometimes, we can convince the network team to use AWS VPC default DNS, but that will introduce another dilemma that the resources in CDP VPC will not be able to resolve the other resources in the custom private domain. Users can create an AWS Route 53 resolver with outbound endpoint and forward the requests to the custom private domain to the custom private DNS server.     ... View more