How to migrate Azure Postgres DB when backend Azure subscription is out of capacity.

[#1]

Requirement introduction

Azure PostgresDB is a PaaS hosted on Azure. Azure hosts PaaS services in an Azure backend subscription. The resources created in the customer’s subscription are actually descriptions and metadata of the actual compute resources in the Azure backend subscription. 

When the Azure backend subscription reaches its capacity limit, Azure will create new resources in a new Azure backend subscription. However, existing PaaS resources will remain in the old Azure backend subscription. When a CDP  user stops a CDP environment, the compute resources for the PostgresDB will be deallocated from the backend Azure subscription, and have to compete for compute resources when being restarted. Suppose it cannot get resources from the backend Azure subscription, as it is already out of capacity. In that case, it cannot be started, and consequently blocks the environment from being started. 

The resolution provided by Microsoft is to back up and restore the database. The restored Postgres DB will allocate the compute resource from a new Azure backend subscription. 

To keep the CDP environment working, the restored DB must have the same database name as the original database. 

High-Level Steps:

1. Stop the environment.
2. Take an ondemand backup of each Azure PG DB and restore the DB to a new PG DB name. (e.g., restored-db)
3. Take ondemand backup of newly created DBs, restored-db 
4. Delete the original PG DBs
5. Restore the PG DBs to the original names from the new backup of the restored-db.
6. Adjust for HA. (Note the original HA configuration, as they may not be the same across)
7. Start the environment.

Detailed Steps:

1. Backup Existing Server 

1. From Cloudera Management Console → Environments (select yours) → Data Lake tab → Database (copy value)
2. Save this value to a notepad on your computer.
3. Repeat the process for Datahub Postgres DB instances and COD instances.

2. Turn off the cluster (and associated datahubs). Do not proceed until it’s off.

1. Locate the same Postgres DB in the Azure Portal, and start the database.
2. With the server ON, navigate to Settings >> Backup and restore and select Backup. (gets the latest backup). Initiate an ondemand FULL backup.
   Note: None of the existing backups can be used.
3. With the server ON, navigate to Settings >> Backup and restore and select Backup. (gets the latest backup). Initiate an ondemand FULL backup.
   
   Note: None of the existing backups can be used.
4. Verify completion (will take several mins):
   
   IMPORTANT! Verify HA config (Check for Zone Redundant or Same Zone)

3. Perform DB “Restore” to create Temporary Database

1. Select the Restore icon on the Datalake DB.
2. Select Fast restore point (Restore using full backup only) – Use the latest one we just took!
   
   
3. Add restored-db to the server name syntax. (Each restore name should be identifiable to map back to the dbsvr-... original name. If the DB name is already taken, change it to a different name.
4. Then make sure networking aligns with the existing PostgreS DB.
   
5. Verify and create.
   
6. Verify completion
   
7. Repeat step 2 process for Datahub Postgres DB instances.
8. Backup NEW Server to create an Ondemand Backup
9. Create ondemand backup for this new database:
   
10. Repeat step 3 process for Datahub Postgres DB instances.

ONLY AFTER COMPLETION OF THIS STEP MOVE FORWARD (Verify successful backup!)

4. Delete OLD DB Server

Repeat step 4 process for Datahub Postgres DB instances.

5. Restore the NEW server to the OLD server name.

1. Important! Make sure to use the same server name the original database had. See example:
   
   
2. Make sure the original network settings copy over:
   
3. Verify completion:
   

Repeat step 5 process for Datahub Postgres DB instances.

6. High Availability

Between backups/restores, HA is turned off, so check for High Availability turned on. Certain Postgres DBs get assigned Zone Redundant, and others are assigned Same Zone so pay attention to original config.

 Verify deployment completion:

 

Repeat step 6 process for Datahub Postgres DB instances.

7. Restart the CDP environment (and clean up)

1. Restart the CDP environment and run basic sanity checks on Cloudera Manager services.
   
   
2. After verifying successful completion, delete the temporary Postgres DB. Repeat this step for any other temporary datahub Postgres DB as well.