Created on 04-15-202102:59 PM - edited on 04-19-202103:48 AM by subratadas
This post covers the steps required to build a custom runtime for Cloudera Data Engineering (CDE). The process pulls the base image from container.repository.cloudera.com and builds a custom image based on the Dockerfile provided and uploads the custom image to Amazon ECR using AWS CodeBuild. All the files mentioned in this post can be downloaded from here.
Setup ECR & IAM role using AWS CloudFormation
The CloudFormation template cloudformation-ecr-codebuild.yml creates the Amazon Elastic Container Repository and the IAM role required for AWS CodeBuild
Update the files cloudformation-ecr-codebuild.yml, cloudformation-parameters.json, and cloudformation-tags.json as required
Create the Cloudformation stack using the following command
Update the Dockerfile with the required customization required to the base image. The base image for the docker file will be provided by the aws-codebuild.json file.
Modify the aws-codebuild.json file with the updates for the environment.
The parameters SOURCE_REPO_USERNAME and SOURCE_REPO_PASSWORD in the aws-codebuild.json file specify the AWS Secrets Manager secret. You can embed the username and password here (or) specify the secret name here.
Quickly review the buildspec.yml file and make modifications if necessary. In general, no changes will be needed here.
Build the custom image using AWS CodeBuild
Zip up the Dockerfile & buildspec.yml (with no root directory) and upload them to the s3 bucket specified in aws-codebuild.json.
Create the CodeBuild project using the following command: