Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)

Objective

The objective of this tutorial is to walk you through process of updating your Cloudbreak Deployer on Amazon AWS from 1.3 to 1.4. Once this is complete, you can now deploy an HDP 2.5 TP cluster.

Prerequisites

  • You should have deployed Cloudbreak 1.3 on Amazon AWS using the instructions found here: Cloudbreak Documentation 1.3 - AWS.
  • You should add TCP port 443 to the security group on Amazon AWS as Cloudbreak 1.4+ appears to proxy requests from port 443 now.
  • Do not run cbd start.

Note: This process will update the Cloudbreak Deployer to the latest available version. At the time of initial writing, this was 1.4. As of September 1, 2016, the latest version is 1.6.

Steps

1. Connect to your Cloudbreak Amazon AWS instance

You should have access to your key file from amazon. Log into your Cloudbreak deployer instance using:

ssh -i <amazon key file> cloudbreak@<amazon instance public ip>

Note: If you have permission issues connecting via ssh, make sure you set your key file permissions to 0600

2. All commands should be run from the Cloudbreak Deployer root directory:

cd /var/lib/cloudbreak-deployer

3. Before you can start Cloudbreak, you need to initialize the environment by running cbd init. You should see something similar to:

$ cbd init
===> Deployer doctor: Checks your environment, and reports a diagnose.
uname: Linux ip-172-31-15-160.ec2.internal 3.10.0-327.10.1.el7.x86_64 #1 SMP Sat Jan 23 04:54:55 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
local version:1.3.0
latest release:1.3.0
docker images:
docker command exists: OK
docker client version: 1.9.1
docker client version: 1.9.1
ping 8.8.8.8 on host: OK
ping github.com on host: OK
ping 8.8.8.8 in container: OK
ping github.com in container: OK

Note: If you previously ran cbd start, then you should run cbd kill before upgrading Cloudbreak.

4. If the initialization completed successfully, now you can update Cloudbreak to version 1.4 using cbd update master && cbd regenerate && cbd pull-parallel. You should see something similar to:

$ cbd update master && cbd regenerate && cbd pull-parallel
Update /usr/bin/cbd from: https://1655-32450069-gh.circle-artifacts.com/0//tmp/circle-artifacts.VMI9cYT/cbd-linux.tgz
mv: try to overwrite '/usr/bin/cbd', overriding mode 0755 (rwxr-xr-x)? y
* removing old docker-compose binary
* Dependency required, installing docker-compose 1.7.1 ...
Generating Cloudbreak client certificate and private key in /var/lib/cloudbreak-deployment/certs.
generating docker-compose.yml
/tmp/bashenv.793850575: line 674: /var/lib/cloudbreak-deployment/.deps/tmp/uaa-delme.yml: No such file or directory
diff: /var/lib/cloudbreak-deployment/.deps/tmp/uaa-delme.yml: No such file or directory
renaming: uaa.yml to: uaa-20160817-135202.yml
generating uaa.yml
latest: Pulling from catatnight/postfix
d64336e52f9a: Pulling fs layer
be760b6bdfc8: Pulling fs layer
2bed4b6dfef0: Pulling fs layer
d64336e52f9a: Downloading 3.784 MB/67.5 MB
be760b6bdfc8: Download complete
2bed4b6dfef0: Download complete
d64336e52f9a: Downloading 11.32 MB/67.5 MB
9e3e55be1c1f: Download complete
089c741cddd5: Download complete
d64336e52f9a: Downloading  12.4 MB/67.5 MB
0d03ba7124d6: Download complete
7ab5db5b9418: Downloading  5.93 MB/20.09 MB
d0f7cd1223d0: Downloading 1.506 MB/14.72 MB
a7aff224adfb: Download complete
7ab5db5b9418: Downloading 14.66 MB/20.09 MB
d0f7cd1223d0: Downloading 10.67 MB/14.72 MB
Digest: sha256:028b5f6f49d87a10e0c03208156ffedcef6da2e7f59efa8886640ba15cbe0e69
7ab5db5b9418: Downloading 15.52 MB/20.09 MB
d0f7cd1223d0: Downloading 11.55 MB/14.72 MB
Digest: sha256:87cf35f319f40f657a68e21e924dd5ba182d8253005c86a116992f2e17570765
Status: Image is up to date for gliderlabs/registrator:v5
v1.0.0: Pulling from library/traefik
cb056548b9bb: Pulling fs layer
1.0.0: Pulling from sequenceiq/socat
1.2.0: Pulling from sequenceiq/cbdb
0ec8b08ed2db: Pulling fs layer
e052a65b2e55: Pulling fs layer
462cae4bc514: Pulling fs layer
f0adfc577336: Pulling fs layer
1.4.0: Pulling from hortonworks/cloudbreak-web
88d2a45c4dd8: Pulling fs layer
d66b5ca0a6a8: Pulling fs layer
d30d76b6140e: Pulling fs layer
a339cf4fec1c: Pulling fs layer
1b32f6c5164a: Pulling fs layer
dd079b923689: Pulling fs layer
ef55927d23dc: Pulling fs layer
09097ea804bd: Pulling fs layer
ff9a6f48abb7: Pulling fs layer
Digest: sha256:8e2ec7a47b17ff50583e05224ca1243ed188aff8087bb546e406effb82b691fe
Status: Image is up to date for sequenceiq/socat:1.0.0
1.4.0: Pulling from hortonworks/cloudbreak-auth
88d2a45c4dd8: Pulling fs layer
17611781a601: Pulling fs layer
9777e7c06cfb: Pulling fs layer
d64336e52f9a: Downloading 15.65 MB/67.5 MB
033171f95048: Pulling fs layer
7ab5db5b9418: Downloading 16.37 MB/20.09 MB
d0f7cd1223d0: Downloading 11.99 MB/14.72 MB
3311ad4fbcb0: Pulling fs layer
1e870254f4fa: Pulling fs layer
13d69d98d1f7: Pulling fs layer
d64336e52f9a: Downloading 16.73 MB/67.5 MB
7ab5db5b9418: Download complete
d0f7cd1223d0: Download complete
d64336e52f9a: Downloading 17.27 MB/67.5 MB
ad3025da7360: Pulling fs layer
v2.7.1: Pulling from sequenceiq/uaadb
1.4.0: Pulling from sequenceiq/periscope
d34921bc2709: Pulling fs layer
7062b3d97728: Pulling fs layer
767584930cea: Pulling fs layer
c05d09cea848: Pulling fs layer
597fe94dd548: Pulling fs layer
Digest: sha256:270e87a90add32c69d8cb848c7455256f4c0a73e14a9ba2c9335b11853f688a6
Status: Image is up to date for sequenceiq/uaadb:v2.7.1
2.7.1: Pulling from sequenceiq/uaa
1.1: Pulling from sequenceiq/haveged
1.4.0: Pulling from sequenceiq/cloudbreak
d34921bc2709: Pulling fs layer
7062b3d97728: Pulling fs layer
767584930cea: Pulling fs layer
c05d09cea848: Pulling fs layer
d64336e52f9a: Downloading 17.81 MB/67.5 MB
c277e7f5e8b7: Pulling fs layer
a30f653c4d56: Pulling fs layer
d64336e52f9a: Downloading 18.35 MB/67.5 MB
7f0c2637ebf6: Pulling fs layer
9e1df59c970a: Pulling fs layer
d64336e52f9a: Downloading 19.97 MB/67.5 MB
6f980013dd43: Pulling fs layer
5f32c66af8ea: Pulling fs layer
eacee569f539: Pulling fs layer
a4c72beb2675: Pulling fs layer
1.2.0: Pulling from sequenceiq/pcdb
Digest: sha256:a64d40d0d51b001d2e0cb8490fcf04da59e0c8ede5121038a175d9bf2374cb6a
Status: Image is up to date for sequenceiq/haveged:1.1
0123c5510cfa: Pulling fs layer
Digest: sha256:361163496cde9183235355b6d043908c96af57a56db4b7d7b2cf40e255026716
Status: Image is up to date for sequenceiq/uaa:2.7.1
c277e7f5e8b7: Pulling fs layer
447edeb914d3: Pulling fs layer
e75814ea06f9: Pulling fs layer
6b4d47a92a9b: Pulling fs layer
Extracting 11.01 MB/18.53 MBExtracting 11.01 MB/18.53 MB
35cab74c8aa7: Downloading 8.466 MB/10.42 MB
c05d09cea848: 71 MB/42.5 MB
c05d09cea848: Downloading 42.18 MB/42.5 MB
Downloading 16.71 MB/18.53 MBownloading 8.623 MB/108.1 MB
d34921bc2709: Pull complete
7062b3d97728: Pull complete
767584930cea: Pull complete
767584930cea: Pull complete
c05d09cea848: Download complete
c05d09cea848: Pull complete
597fe94dd548: Pull complete
c277e7f5e8b7: Pull complete
c277e7f5e8b7: Pull complete
Downloading 50.25 MB/108.1 MB07 MB/130.1 MB
6b4d47a92a9b: Downloading  5.14 MB/10.42 MB
Downloading 2.741 MB/10.42 MBownloading 7.001 MB/13.19 MB
a30f653c4d56: Pull complete
Extracting 117.5 MB/130.1 MBExtracting 117.5 MB/130.1 MB
5cb6cc1fb08d: Pull complete
eacee569f539: Extracting 284.4 kB/284.4 kB
d66b5ca0a6a8: Pull complete
Extracting 16.Extracting 24.71 MB/42.5 MB
7f0c2637ebf6: Pull complete
9e1df59c970a: Pull complete
42396e8dcbcd: Pull complete
6f980013dd43: Extracting     32 B/32 B
6f980013dd43: Extracting     32 B/32 B
6f980013dd43: Pull complete
5f32c66af8ea: Pull complete
eacee569f539: Pull complete
a4c72beb2675: Pull complete
447edeb914d3: Pull complete
e75814ea06f9: Pull complete
6b4d47a92a9b: Pull complete
4b4f74f41ebf: Pull complete
be602741d584: Pull complete
07b4015931e0: Pull complete
5cab237e98a9: Pull complete
3a12055ee388: Extracting     32 B/32 B
73fb6d32d5e3: Pull complete
3a12055ee388: Pull complete
18afddb9bf55: Pull complete
Digest: sha256:8085718c474c40ce4dcc5f64b9ccf23a3f91b3cb2f7fe2e8572fc549a25e6953
Status: Downloaded newer image for sequenceiq/cloudbreak:1.4.0

5. Once the upgrade process is complete, start Cloudbreak using cbd start. You should see something similar to:

$ cbd start
generating docker-compose.yml
generating uaa.yml
Creating cbreak_haveged_1...
Creating cbreak_uluwatu_1...
Creating cbreak_cbdb_1...
Creating cbreak_consul_1...
Creating cbreak_cloudbreak_1...
Creating cbreak_registrator_1...
Creating cbreak_pcdb_1...
Creating cbreak_periscope_1...
Creating cbreak_sultans_1...
Creating cbreak_uaadb_1...
Creating cbreak_logsink_1...
Creating cbreak_logspout_1...
Creating cbreak_identity_1...
Uluwatu (Cloudbreak UI) url:
  http://54.164.138.139:3000
login email:
  admin@example.com
password:
  cloudbreak

6. Login to the Cloudbreak UI.

Note: As I mentioned in the prerequisites, Cloudbreak appears to proxy requests from port 443 now. The url to access the Cloudbreak UI will be https://<amazon cloudbreak instance ip> As of Cloudbreak 1.6, the properly URL is displayed for the UI.

7. Create a platform definition. This is done by expand the manage platforms area of the Cloudbreak UI. We are using AWS, so create a platform by selecting AWS. The UI will look similar to this:

6774-manage-platforms-1.png

You can provide any Name and Description you like.

8. Create a credential. This is done by expand the manage credentials area of the Cloudbreak UI. We are using AWS, so select AWS. For ease of configuration, change the AWS Credential Type to Key Based. The Select Platform option should be set to the platform you created in the previous step. The UI will look similar to this:

6775-manage-credentials.png

You can provide any Name and Description you like. The Access Key and Secret Access Key are from your Amazon account. See this documentation to setup an access key: AWS Credentials. The SSH Public Key is found in the /home/cloudbreak/.ssh/id_rsa.pub file that was created when you followed the steps for creating the Cloudbreak instance. You can see the key by using cat on the file like this:

$ cat /home/cloudbreak/.ssh/id_rsa.pub

Note: Remember to download your credentials from Amazon. If you forget this step, there is no way to determine your Secret Access Key. You will have to delete those credentials and create new ones.

9. Once the credential is created, you need to select it.

6776-select-credentials.png

10. Now we will create our own blueprint by copying one of the existing ones. We are doing this to deploy HDP 2.5. The default blueprints will currently deploy HDP 2.4.

Note: As of Cloudbreak 1.6, the default version is HDP 2.5 so this step is not necessary.

Expand the manage blueprints section of the UI. The UI will look similar to this:

6782-manage-blueprints-1.png

Now select the hdp-small-default blueprint. We will copy this for our HDP 2.5 blueprint. The UI will look similar to this:

6783-manage-blueprints-2.png

Click the copy & edit button to create a copy of the blueprint. The UI will look similar to this:

6784-manage-blueprints-3.png

You can provide any Name and Description that you like. In the JSON Text field, scroll down to the bottom. Change the "stack.version": "2.4" to "stack.version": "2.5". Click the green create blueprint button.

11. Now you can create your cluster. click the green Create cluster button.

6777-create-cluster-1.png

Provide a Cluster Name and select the appropriate AWS Region. Click the Setup Network and Security button.

6778-create-cluster-2.png

You don't need to change anything here. Click the Choose Blueprint button.

6779-create-cluster-3.png

Select the Blueprint we created in the previous steps. You will notice there is an Ambari Server check box on each of the servers listed. You need to determine where you want to deploy Ambari. Select the checkbox for that server. The UI will look similar to this:

6780-create-cluster-4.png

Click the Review and Launch button. This will provide a final confirmation screen with a summary of the cluster.

6781-create-cluster-5.png

If everything looks good, click the green create and start cluster button.

12. Cloudbreak will now start creating the cluster. The UI will look similar to this:

6786-create-cluster-progress-1.png

13. If you click on the test1 cluster name, you can see more information on the cluster. The UI will look similar to this:

6787-create-cluster-progress-2.png

14. Once the cluster build is complete, the UI should look similar to this:

6788-create-cluster-progress-3.png

You can see more information during the cluster build process by expanding the Event History section. The UI will look similar to this:

6789-create-cluster-progress-4.png

15. Once the cluster build is complete, you can log into Ambari using the Ambari Server Address link provided.

16.Once you are logged in to Ambari, select the Stacks and Versions view.

6790-ambari-1.png

17. You can see by the components listed there are new HDP 2.5 components like Log Search and Spark2. You should see something similar to:

6791-ambari-2.png

18 And finally, you can see HDP version by click the Versions tab. You should see something similar to:

6792-ambari-3.png

Review

We successfully upgraded Cloudbreak 1.3 to Cloudbreak 1.4 on Amazon AWS. Using Cloudbreak 1.4, we were able to clone a blueprint, change the stack version to 2.5 and deploy a HDP 2.5 TP cluster.

955 Views
Comments
Not applicable

Michael Young, Awesome steps with snap......Please suggest what would be the minimum size of the cluster supported either by aws or gcp? What would be the type/size of machine would be idle?

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 10:46 AM
Updated by:
 
Contributors
Top Kudoed Authors