Migrate CDE jobs from one virtual cluster to another

[#1]

Introduction

In Cloudera Data Engineering (CDE), you may have a need to move the jobs from one virtual cluster to another. This can be for testing purposes or to move the jobs into the latest CDE versions. This can be easily achieved by the backup & restore process and is well documented here. The purpose of this article is to provide additional tools to simplify this and demonstrate the process. The example shown here demonstrates this process within the same CDE service, but you can use the same approach to move across CDE services.

Steps

1. Ensure that CDP CLI is set up on the system with the required access to list and describe CDE services & virtual clusters. Also, have your workload password handy as this is required for the export and import process.
2. Download export & import scripts using the following commands.
   

   wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-cde/main/export-import-cde-jobs/cde-vc-jobs-export.sh
   wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-cde/main/export-import-cde-jobs/cde-vc-jobs-import.sh​​

3. In this example, we will move the jobs from the virtual cluster sourceVC to targetVC. The sourceVC contains three jobs and the targetVC does not contain any job.
   
   
   
   
   
4. Run the following command to export the job from sourceVC. When prompted, enter the workload password. Make a note of the backup zip file from the output.
   

   Usage: ./cde-vc-jobs-export.sh <source-cluster-name> <source-vc-name> <workload-username>​​

   

5. Run the following command to import the job from targetVC. When prompted, enter the workload password.
   

   Usage: ./cde-vc-jobs-import.sh <target-cluster-name> <target-vc-name> <workload-username> <file-name>​​

   

   Here is a view of the targetVC after the successful import.
   You can use more advanced options listed here to export jobs based on job names, include resources, etc. Modify the script as per your requirement.