Cloudbreak 2.9.1 maintenance release is now available. If you are using an earlier version of Cloudbreak, you can upgrade now to pick up the latest bug fixes. If you are new to Cloudbreak, you can get started by launching Cloudbreak on AWS, Azure, GCP, or OpenStack from a template. Useful links: Release notes Upgrade steps Get started on AWS Get started on Azure Get started on GCP ... View more

If you would like to upgrade Cloudbreak in an environment with no internet access, refer to the following documentation: https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/upgrade/content/cb_upgrade-cloudbreak-no-internet.html ... View more

If you would like to configure Cloudbreak to use an external database that uses SSL, refer to the following documentation: https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/configure/content/cb_configure-external-ssl.html ... View more

Cloudbreak 2.9.0 is now available! It is a general availability (GA) release, so - with an exception of some features that are marked as TP - it is suitable for production. Try it now Upgrade to 2.9.0 Quickly deploy by using quickstart on AWS, Azure, or Google Cloud Install manually on AWS, Azure, Google Cloud, or OpenStack New features Cloudbreak 2.9.0 introduces the following new features. While some of these features were introduced in Cloudbreak 2.8.0 TP, others are brand new: Feature Description Documentation Specifying resource group name on Azure When creating a cluster on Azure, you can specify the name for the new resource group where the cluster will be deployed. Resource group name Multiple existing security groups on AWS When creating a cluster on AWS, you can select multiple existing security groups. This option is available only when an existing VPC is selected. Create a cluster on AWS EBS volume encryption on AWS You can optionally configure encryption for EBS volumes attached to cluster instances running on EC2. Default or customer-managed encryption keys can be used. EBS encryption on AWS Shared VPCs on GCP When creating a cluster on Google Cloud, you can place it in an existing shared VPC. Shared networks on GCP GCP volume encryption By default, Google Compute Engine encrypts data at rest stored on disks. You can optionally configure encryption for the encryption keys used for disk encryption. Customer-supplied (CSEK) or customer-managed (CMEK) encryption keys can be used. Disk encryption on GCP Workspaces Cloudbreak introduces a new authorization model, which allows resource sharing via workspaces. In addition to a default personal workspace, users can create additional shared workspaces. Workspaces Operations audit logging Cloudbreak records an audit trail of the actions performed by Cloudbreak users as well as those performed by the Cloudbreak application. Operations audit logging Updating long-running clusters Cloudbreak supports updating base image's operating system and any third party packages that have been installed, as well as upgrading Ambari, HDP and HDF. Updating OS and tools on long-running clusters and Updating Ambari and HDP/HDF on long-running clusters HDP 3.1 Cloudbreak introduces two default HDP 3.1 blueprints and allows you to create your custom HDP 3.1 blueprints. Default cluster configurations HDF 3.3 Cloudbreak introduces two default HDF 3.3 blueprints and allows you to create your custom HDP 3.3 blueprints. To get started, refer to How to create a NiFi cluster HCC post. Default cluster configurations Recipe parameters Supported parameters can be specified in recipes as variables by using mustache kind of templating with "{{{ }}}" syntax. Writing recipes and Recipe parameters Shebang in Python recipes Cloudbreak supports using shebang in Python scripts run as recipes. Writing recipes Technical preview features The following features are technical preview (not suitable for production): Feature Description Documentation AWS GovCloud (TP) You can install Cloudbreak and create Cloudbreak-managed clusters on AWS GovCloud. Deploying on AWS GovCloud Azure ADLS Gen2 (TP) When creating a cluster on Azure, you can optionally configure access to ADLS Gen2. This feature is technical preview. Configuring access to ADLS Gen2 New and changed data lake blueprints (TP) Cloudbreak includes three data lake blueprints, two for HDP 2.6 (HA and Atlas) and one for HDP 3.1. Note that Hive Metastore has been removed from the HDP 3.x data lake blueprints, but setting up an external database allows all clusters attached to a data lake to connect to the same Hive Metastore. To get started with data lakes, refer to How to create a data lake with Cloudbreak 2.9 HCC post. Working with data lakes Default blueprints Cloudbreak 2.9.0 includes the following HDP 2.6, HDP 3.1, and HDF 3.3 workload cluster blueprints. In addition, HDP 3.1 and HDP 2.6 data lake blueprints are available as technical preview. Note that Hive Metastore has been removed from the HDP 3.x data lake blueprints, but setting up an external database allows all clusters attached to a data lake to connect to the same Hive Metastore. Documentation links How to create a data lake with Cloudbreak 2.9 (HCC post) How to create a NiFi cluster (HCC post) Cloudbreak 2.9.0 documentation (Official docs) Release notes (Official docs) ... View more

Cloudbreak 2.7.2 maintenance release is now available. If you are using an earlier version of Cloudbreak, you can upgrade now to pick up the latest bug fixes. If you are new to Cloudbreak, you can get started by launching Cloudbreak on AWS, Azure, GCP, or OpenStack from a template. Useful links: Release notes Upgrade steps Get started ... View more

In the following video, I demonstrate how to create a Cloudbreak credential on Google Cloud Video link: https://youtu.be/uVYpgz9m4eE The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the roles that need to be assigned to the service account, refer to Service account for GCP credential documentation. If you are using a corporate Google Cloud account you may be unable to perform some of the steps (such as service account creation and role assignment) by yourself and you may have to contact your Google Cloud admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more

In the following video, I demonstrate how to create a key-based Cloudbreak credential on AWS Video link: https://youtu.be/SIYhzlMQf8Y The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the policy that need to be assigned to the user, refer to Create CredentialRole documentation. If you are using a corporate AWS account you may be unable to perform some of the steps (such as user creation and policy assignment) by yourself and you may have to contact your AWS admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more

In the following video, I demonstrate how to create an app-based Cloudbreak credential on Azure Video link: https://youtu.be/NYQZd2AmM6g The Cloudbreak version used in this video is Cloudbreak 2.7.1. If you are using a corporate Azure account you may be unable to perform some of the steps (such as role creation and policy assignment) by yourself and you may have to contact your Azure admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more

In the following video, I demonstrate how to create an interactive Cloudbreak credential on Azure Video link: https://youtu.be/Q9fqmOZqnnw The Cloudbreak version used in this video is Cloudbreak 2.7.1. If you are using a corporate Azure account, you may be unable to use this method due to the fact that the Owner role is required. In this case, you should use the app-based credential method. See Create an app-based credential documentation. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more

In the following video, I demonstrate how to create a role-based Cloudbreak credential on AWS Video link: https: / /youtu.be /XlvUahqwThI The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the policy that need to be assigned to the user, refer to Create CredentialRole documentation. If you are using a corporate AWS account you may be unable to perform some of the steps (such as role creation and policy assignment) by yourself and you may have to contact your AWS admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more

I found the following statement in the docs related to creating the IAM role with the AssumeRole policy: “Alternatively, instead of attaching the “CloudbreakRole” role during the VM launch, you can assign the“CloudbreakRole”to an IAM user and then add the access and secret key of that user to your ‘Profile’.” It appears here: https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.1/content/aws-pre/index.html#option-2-configure-role-based-authentication This statement seems incorrect to me since it is not possible to attach roles to users. You can only attach policies to users. Maybe this was supposed to say “you can assign the“Assume Role” policy to an IAM user and then add the access and secret key of that user to your ‘Profile’.” Does anyone remember how this work? I think I copied this from the old Cloudbreak docs. Never tried it. Thanks! ... View more

Starting with HDP 2.6.5, it is possible to configure access to Google Cloud Storage from HDP via the gs cloud storage connector. There are two ways to do this: - You can use Cloudbreak to deploy clusters on GCP and help you configure access to GCS from your Cloudbreak-managed HDP cluster. The steps involve (1) Setting up a service account on Google Cloud and (2) Providing the Service Account Email Address to Cloudbreak in the cluster create wizard on the Cloud Storage page. For detailed steps, see https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.1/content/gcp-data/index.html - It is also possible to configure access to GCS by setting up a service account, manually placing the service account key on all nodes of the cluster, and setting related properties in Ambari. For detailed steps, see https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/bk_cloud-data-access/content/authentication-gcp.html Note that this capability is currently only available as technical preview. ... View more

Error: Cloudbreak cluster fails with "Infrastructure creation failed. Reason: Failed to retrieve the server's certificate Solution: The most common reason for this error is that your security group configuration is too restrictive. Notably, CB needs access to port 9443 from outside the local network. Refer to https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.0/content/security/index.html#default-cluster-security-groups for information on ports that must be open on cluster security groups. ... View more

Error: Cloudbreak cluster fails with "Infrastructure creation failed. Reason: Failed to retrieve the server's certificate Solution: The most common reason for this error is that your security group configuration is too restrictive. Notably, CB needs access to port 9443 from outside the local network. Refer to https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.0/content/security/index.html#default-cluster-security-groups for information on ports that must be open on cluster security groups. ... View more

Cloudbreak 2.4.2 has been released. In addition to a few bug fixes, the release comes with new base and prewarmed images including Ambari 2.6.2 and HDP 2.6.5 (Cloudbreak 2.4.0 and 2.4.1 came with earlier Ambari and HDP versions). For documentation, see https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.2/content/index.html For release notes, see https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.2/content/releasenotes/index.html ... View more

Cloudbreak 2.4.0 and newer includes a neat feature which allows you to obtain CLI commands from the web UI. This feature can save you a lot of time if you are just getting started with the CLI. Prerequisites You must have an instance of Cloudbreak running. You downloaded and configured Cloudbreak CLI. if you still need to do this, refer to Installing the CLI. Obtain Cluster JSON from the UI By far the most useful feature is being able to obtain cluster JSON file from the web UI. There are two easy ways to do this. Option 1 (Does not require creating a cluster) Navigate to the create cluster wizard and specify all required parameters. Once done, on the last page of the wizard there is an option to SHOW CLI COMMAND, which allows you to generate a JSON skeleton of your cluster: You can click COPY the JSON, paste it to a text edit, and save it as template.json. Next, you can copy the command listed in the web UI below the CLI skeleton to create a cluster from the template.json file: cb cluster create --cli-input-json template.json --name cli-cluster Option 2 (Requires that you create a cluster) Another way to obtain cluster JSON skeleton is by navigating to the details of a cluster that has already been created. Under ACTIONS, there is an option to SHOW CLI COMMAND, which allows you to generate a JSON skeleton of your cluster: Obtain CLI Command Syntax from the UI In addition to allowing you to obtain CLI JSON content from the web UI, Cloudbreak allows you to obtain some commands from the UI by using the SHOW CLI COMMAND option. For example: When creating a credential, you can generate `cb credential create` When creating a blueprint, you can generate `cb blueprint create` When adding a recipe, you can generate `cb recipe create` ... View more

Due to a recent change in Google Cloud's handling of requests, all clusters created on Google Cloud Platform with Cloudbreak Cloudbreak 2.4.0 (or earlier 2.x) and 1.16.5 (or earlier 1.x) fail with an error similar to: Infrastructure creation failed. Reason: com.sequenceiq.cloudbreak.cloud.gcp.GcpResourceException: Error during status check: [ resourceType: GCP_DISK, resourceName: hdf-test, stackId: 5, operation: hdftest-s-2-20180323123311 ] [ Cause message: 403 Forbidden { "code" : 403, "errors" : [ { "domain" : "global", "message" : "Required 'compute.globalOperations.get' permission for 'projects/*******/global/operations/operation-1521808391647-56813a099af18-300f33eb-8a28c20c'", "reason" : "forbidden" } ], "message" : "Required 'compute.globalOperations.get' permission for 'projects/*******/global/operations/operation-1521808391647-56813a099af18-300f33eb-8a28c20c'" } ] In order to address the issue, we created a 2.4.1 maintenance release. You can upgrade to Cloudbreak 2.4.1 by using cbd update. For detailed update steps, refer to Cloudbreak 2.4.1 docs. > As a side note, Cloudbreak 2.4.1 also includes a fix for time-based scaling. If you are using Cloudbreak 1.x and do not want to update to Cloudbreak 2.x, you can update to Cloudbreak 1.16.6 by following these steps: 1.Download the CBD binary: curl -Ls public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_1.16.6_$(uname)_x86_64.tgz | sudo tar -xz -C /bin cbd 2.After updating, restart Cloudbreak (must be done from the deployment directory): cbd restart ... View more