How-To: SSH Key Rotation Process for Environments, Data Lake and Data Hub on Public Cloud

[#1]

Prerequisites

Before beginning the key rotation process, ensure the following are in place:

• Virtual machine account access or the ability to create them via ssh-keygen: You must have the necessary permissions to manage Virtual Machines and Key Vaults in your cloud provider environment or via bash command line interface.
• New SSH key pair: Generate a new, secure SSH key pair (public and private).

Assumptions

1. You have an old key called id_rsa and id_rsa.pub.
2. You have generated a new key pair, either via your cloud provider SSH Keys service or via ssh-keygen via Bash CLI. 
3. We will assume; for the purpose of this guide, that the new keys are id_rsa_new and id_rsa_new.pub.
4. You have access to the Cloudera Control Plane. 
5. You have PowerUser or similar permissions to update the Summary tab values in your environment.

1. Generate a New SSH Key Pair

A new, unique key pair must be generated for the rotation.

Key Step	Command (Linux/macOS)	Description
Generate key	ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_new	Creates a new 4096-bit RSA key pair.
Secure Private Key	chmod 400 ~/.ssh/id_rsa_new	Restricts access to the private key.
Get Public Key	cat ~/.ssh/id_rsa_new.pub	Displays the new public key contents.

2. Update the new SSH Public Key in the Summary tab of the Cloudera Environment

The new public key must be added to the existing environment under the “Summary” tab in the “Advanced” box at the end of the page.

1. Retrieve the public key string: Copy the full output generated by the cat command (for example, ssh-rsa AAAA...user@host).
2. Modify the key in the Console: Click the edit icon located in the top-right corner to update the field with your new key.

IMPORTANT: A notification will appear stating that the update only affects new Data Hub deployments. While accurate for initial behavior, this guide later details how to push this updated key to your active FreeIPA, Data Lake, and Data Hub nodes.

3. Rotating the new Public SSH key to FreeIPA

Before rotating the old key for the new key, verify that the old key pair grants successful access to the FreeIPA nodes.

1. Test SSH connectivity: Attempt to connect to the virtual machine using the old private key. 

   ssh -i ~/.ssh/id_rsa cloudbreak@<freeipa_ip_address>

2. Once  you have verified the old key still works, in order to rotate the old key and add the new SSH key to the FreeIPA nodes, you will need to navigate to “FreeIPA > Security > Search for “Cloudbreak user root SSH public key” > Check the box > Click on Rotate Secrets”.
3. This will effectively trigger a Salt command behind the scenes that will propagate this new key to all of the FreeIPA nodes. I assume this would be something like the command below or a more complex workflow.

Verification

1. Once this process is triggered, you will see that the FreeIPA status will switch to “Update in progress”. Wait for about 10-15 minutes and the process will finish. 
2. Test out the new key and this time you should be able to ssh with your new private key. 

   ssh -i ~/.ssh/id_rsa_new cloudbreak@<vm_ip_address>

This process ensures that even if you have to replace/repair a node in your DL, Environment or Data Hub, the new key will always be used. The old key will not be used anymore. 

4. Rotating the new key to Data Lake nodes

The process here is the same as for the FreeIPA nodes, but in this case, we will have to navigate to the Data Lake tab and from there follow the same route.

1. Test SSH connectivity: Attempt to connect to the VM using the old private key. 

   ssh -i ~/.ssh/id_rsa cloudbreak@<datalake_node_ip_address>

2. Now, you have verified the old key still works, to rotate the old key and add the new SSH key to the Data Lake nodes, you will need to navigate to Data Lake > Security > Search for “Cloudbreak user root SSH public key” > Check the box > Click on Rotate Secrets. This should effectively trigger an update on the Data Lake nodes.

Verification

1. You should now see a new state on the Data Lake which states “Update in progress”. 
2. Check your Data Lake Event History to see what is effectively happening.
3. This process ensures that even if you have to replace/repair a node in your DL, Environment or Data Hub, the new key will always be used. The old key will not be used anymore. 

5. Rotating the new key to Data Hub nodes

The process here is the same as for the FreeIPA and Data Lake nodes, but in this case, we will have to navigate to the Data Hub in question and from there follow the same route.

1. Test SSH connectivity: Attempt to connect to the VM using the old private key.

   ssh -i ~/.ssh/id_rsa cloudbreak@<datahub_node_ip_address>

2. Once  you have verified the old key still works, to rotate the old key and add the new SSH key to the Data Hub nodes, you will need to navigate to “Go to your Data Hub in question > Security > Search for “Cloudbreak user root SSH public key” > Check the box > Click on Rotate Secrets”. This should effectively trigger an update on the Data Hub nodes.

Verification

1. You should now see a new state on the Data Hub which states “Update in progress”. 
2. Check your Data Hub’s Event History to see what is effectively happening.
3. This process ensures that even if you have to replace/repair a node in your DL, Environment or Data Hub, the new key will always be used. The old key will not be used anymore. 

6. Checking that old keys are no longer authorized

For enhanced security, remove the old key from the VM if for some reason the salt process failed to do so. If the VM was provisioned using an older key, it may still be in the authorized keys file. The Salt process should have removed it, but better be safe than sorry! You can also validate this by using the old key and checking if it connects or prompts for a password. If the latter happens, this means the key has been unauthorized and removed from the file.

1. SSH into the virtual machine using the new key.
2. Edit the authorized_keys file:

   sudo vi /home/cloudbreak/.ssh/authorized_keys

3. Delete the old public key entry from the file.
4. Save and close the file.

7. Final Documentation and Cleanup Steps For Customer To Perform

After a successful rotation, complete the following tasks:

Action	Owner	Status
Archive old private key	Person	Pending
Update Key Rotation Log	File	Pending
Schedule next rotation date	Date	Pending
Update access documentation	File	Pending
Notify team of change	Calendar event	Pending

8. Leveraging CDP CLI for Automation

Automating the rotation workflow is feasible through the CDP CLI. To implement this, follow these structural steps: 

1. Initiate the update of the Environment's SSH Public Key via the update-ssh-key command. Reference: https://cloudera.github.io/cdp-dev-docs/cli-docs/environments/update-ssh-key.html.
2. Execute secret rotation across Data Lake, FreeIPA, and Data Hubs using their respective APIs:
1. Data Lake: Apply rotate-secrets (see CLI docs). Detailed --secret-types guidance is available here.
2. FreeIPA: Apply rotate-freeipa-secrets (see CLI docs). Detailed --secret-types guidance is available here.
3. Data Hub: Apply rotate-secrets (see CLI docs). Detailed --secret-types guidance is available here.
3. Following these actions, the updated Public Key will propagate to all cluster nodes.
4. REMINDER: CLI-based procedures may vary depending on your specific Cloud Provider.

# Step 1: Update Environment SSH Key

cdp environments update-ssh-key \ 
--environment [ENV-CRN-GOES-HERE] \ 
--new-public-key ["SSH_PUBLIC_KEY_TEXT"] 

# Step 2a: Rotate Data Lake Secrets cdp datalake rotate-secrets \ 
--datalake [DL-CRN-GOES-HERE] \ 
--secret-types "USER_KEYPAIR" 

# Step 2b: Rotate FreeIPA Secrets 
cdp environments rotate-freeipa-secrets \ 
--datahub [DATAHUB-CRN-GOES-HERE] \ 
--secret-types "USER_KEYPAIR" 

# Step 2c: Rotate Data Hub Secrets 
cdp datahub rotate-secrets \ 
--datahub [DATAHUB-CRN-GOES-HERE] \ 
--secret-types "USER_KEYPAIR"