Cloudera Data Platform (CDP) for public cloud has an amazing admin UI that drives you through a thorough wizards for setting up environment, data lakes, data hub clusters and experiences.
Details of AWS pre-requisites can be found in CDP official documentation, but why use the easy button when you can leverage AWS and CDP cli and do it the hard way?
My thoughts exactly.
Here is the TL;DR: go to my github and run the scripts as instructed.
Even through cli, a lot of the configuration of a environment can be automated, including:
However, you will need to set this minimal set of elements to get an environment ready:
To better understand how roles, policies and bucket interact, you can refer to this diagram:
In this tutorial however, I'm not going to worry about additional roles but the dataake_admin_role and ranger_audit_role (the other ones are optional).
Install and configure the AWS cli with your account: link
Create a public/private key pair: link
Create CDP credential for AWS: link
Install and configure CDP CLI: link
Clone my github repository
git clone https://github.com/paulvid/cdp_create_env_aws.git
aws_create_bucket.sh <base_dir> <prefix> <region>
aws_purge_roles_policies.sh <base_dir> <prefix>
aws_create_policies.sh <base_dir> <prefix>
aws_create_roles.sh <base_dir> <prefix> <bucket>
cdp_create_env.sh <base_dir> <prefix> <credential> <region> <key>
Obviously this is just getting us started.
I plan on publishing much more about creating data lakes, data hub clusters and much more. Stay tuned!