Cloudbreak 2.7.2 maintenance release is now available. If you are using an earlier version of Cloudbreak, you can upgrade now to pick up the latest bug fixes. If you are new to Cloudbreak, you can get started by launching Cloudbreak on AWS, Azure, GCP, or OpenStack from a template. Useful links: Release notes Upgrade steps Get started ... View more

Cloudbreak 2.8.0 Technical Preview release is now available! New features Cloudbreak 2.8.0 TP introduces the following new features: Feature Description Documentation AWS GovCloud Cloudbreak supports installing Cloudbreak and creating Cloudbreak-managed clusters on AWS GovCloud. Deploying on AWS vs AWS GovCloud EBS volume encryption on AWS You can optionally configure encryption for EBS volumes attached to cluster instances running on EC2. Default or customer-managed encryption keys can be used. EBS encryption on AWS GCP volume encryption By default, Compute Engine encrypts data at rest stored on disks. You can optionally configure encryption for the encryption keys used for disk encryption. Customer-supplied (CSEK) or customer-managed (CMEK) encryption keys can be used. Disk encryption on GCP User authorization Cloudbreak introduces a new authorization model, which allows resource sharing via organizations. User authorization Operations audit logging Cloudbreak records an audit trail of the actions performed by Cloudbreak users as well as those performed by the Cloudbreak application. Operations audit logging Updating long-running cluster Cloudbreak supports updating base image's operating system and any third party packages that have been installed. Updating long-running clusters Data lake HA and Atlas support Cloudbreak includes two data lake blueprints: Data lake HA blueprint Data lake blueprint including Atlas (HA is not supported) Working with Data Lakes (TP) Multiple existing security groups on AWS Multiple existing security groups can be specified when creating a cluster via CLI on AWS. Multiple existing security groups on AWS Shebang in Python recipes Cloudbreak supports using shebang in Python scripts run as recipes. Writing recipes HDF 3.2 Cloudbreak can be used to deploy HDF 3.2 clusters by using one of the two default HDF 3.2 blueprints: Flow Management clusters with Apache NiFi Messaging clusters with Apache Kafka. Default cluster configurations For more information on what changed in Cloudbreak 2.8.0 TP, refer to Release Notes. Redesigned documentation Starting with Cloudbreak 2.8.0, Cloudbreak documentation is published in the same format (HTML and PDF) as other Hortonworks documentation. You can access the available publications here: https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.8.0/index.html Here is what you should expect to find in each publication: RELEASE NOTES Release Notes: New features, behavioral changes, known issues, fixed issues, and image catalog updates. CONCEPTS Introduction to Cloudbreak: High-level conceptual information about Cloudbreak. Cloudbreak Security Overview: High-level conceptual information related to security in Cloudbreak. INSTALLATION & UPGRADE Cloudbreak Deployment Options: Introduction to Cloudbreak and cluster installation options, helping you decide which option to use. Quickstart on AWS/Azure/GCP: Instructions for how to install Cloudbreak from a template. This is typically not suitable for production. Installing Cloudbreak on AWS/Azure/GCP/OpenStack: Instructions for how to install Cloudbreak on your own VM. This is typically used for production. Upgrading Cloudbreak: Instructions for how to upgrade your Cloudbreak instance. Always refer to upgrade documentation for the Cloudbreak version that you are upgrading to. Installing Cloudbreak CLI: Instructions for installing and configuring Cloudbreak CLI. HOW TO Creating a Cloudbreak Credential on AWS/Azure/GCP/OpenStack: Cloud provider-specific instructions for creating a Cloudbreak credential. Creating a Cluster on AWS/Azure/GCP/OpenStack: Cloud provider-specific instructions for creating HDP and HDF clusters and an overview of advanced cluster options. Accessing Clusters: Information about user accounts and instructions for accessing Cloudbreak-managed clusters. Managing Clusters: Instructions for how to perform cluster management tasks such as resizing, stopping, configuring autoscaling, configuring an SSL certificate, and updating OS on cluster VMs. Advanced Cluster Options: Instructions for how to configure advanced cluster options such as custom images, recipes, Kerberos security, disk encryption, external databases, LDAP/AD, and more. Configuring Access to Cloud Data: Instructions for how to configure access from a cluster created via Cloudbreak to Amazon S3, ADLS, WASB, or GCS. Working with Data Lakes: Introduction to data lakes and data lake setup-steps. Advanced Cloudbreak Configuration: Instructions on how to set up an external Cloudbreak database, LDAP/AD or Cloudbreak, and other advanced Cloudbreak configurations. Managing and Monitoring Cloudbreak: (NEW) Information about Cloudbreak’s user authentication model and operational audit logging. TROUBLESHOOTING Troubleshooting Cloudbreak: Steps for troubleshooting Cloudbreak. Also includes the location of Cloudbreak logs. REFERENCE CLI Reference: Cloudbreak CLI reference, including command examples. Developer Documentation: Links to API and other developer docs.' Get started with Cloudbreak Use the following links to install or upgrade Cloudbreak: Quickstart on AWS/Azure/GCP Upgrading Cloudbreak Cloudbreak Deployment Options ... View more

Although as of Cloudbreak 2.8.0 there is no way to manage cluster templates in the Cloudbreak web UI, you can generate a reusable template from the UI and then reuse it for creating clusters via Cloudbreak CLI. In this blog post, I describe the steps for how to do this. Step 1: In the web UI, provide cluster parameters and generate a cluster template 1.In the Cloudbreak web UI, navigate to Clusters and then click on Create cluster. 2.To obtain a full list of cluster options, click on Advanced. 3.Provide all the parameters for your cluster. 4.Once done, on the last page of the create cluster wizard, click on Show CLI Command. 5.Click on Copy the JSON to copy the generated CLI skeleton. 6.Paste it into a text editor and save the file onto your computer in .JSON format. 7.Open the file and review it to ensure that no parameters are missing. Save the file once you are done editing. Step 2: Create clusters via CLI from the template 1.Download and configure Cloudbreak CLI. Refer to Install CLI and Configure the CLI. 2.To create a cluster from the template generated in the previous step, use the following command: cb cluster create --cli-input-json <path-to-the-cluster-template> For example: cb cluster create --cli-input-json /Users/myuser/Documents/mytemplate.json More options If you would like to create multiple clusters from one template, you will have to provide a different name for each cluster. You can do this by not specifying the cluster name in the template, but passing it with each cb cluster create command. For example: cb cluster create --name mycluster1 --cli-input-json /Users/myuser/Documents/mytemplate.json As an alternative for generating cluster template from the UI, you can use the following command to generate a cluster template: cb cluster generate-template However, when specifying many advanced options (such as custom Ambari and stack repos), I find it easier, faster, and more precise to generate a template from the UI. ... View more

In the following video, I demonstrate how to create a Cloudbreak credential on Google Cloud Video link: https://youtu.be/uVYpgz9m4eE The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the roles that need to be assigned to the service account, refer to Service account for GCP credential documentation. If you are using a corporate Google Cloud account you may be unable to perform some of the steps (such as service account creation and role assignment) by yourself and you may have to contact your Google Cloud admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more

In the following video, I demonstrate how to create a key-based Cloudbreak credential on AWS Video link: https://youtu.be/SIYhzlMQf8Y The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the policy that need to be assigned to the user, refer to Create CredentialRole documentation. If you are using a corporate AWS account you may be unable to perform some of the steps (such as user creation and policy assignment) by yourself and you may have to contact your AWS admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more

In the following video, I demonstrate how to create an app-based Cloudbreak credential on Azure Video link: https://youtu.be/NYQZd2AmM6g The Cloudbreak version used in this video is Cloudbreak 2.7.1. If you are using a corporate Azure account you may be unable to perform some of the steps (such as role creation and policy assignment) by yourself and you may have to contact your Azure admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more