Cloudbreak 2.9.0 is now available! It is a general availability (GA) release, so - with an exception of some features that are marked as TP - it is suitable for production. Try it now Upgrade to 2.9.0 Quickly deploy by using quickstart on AWS, Azure, or Google Cloud Install manually on AWS, Azure, Google Cloud, or OpenStack New features Cloudbreak 2.9.0 introduces the following new features. While some of these features were introduced in Cloudbreak 2.8.0 TP, others are brand new: Feature Description Documentation Specifying resource group name on Azure When creating a cluster on Azure, you can specify the name for the new resource group where the cluster will be deployed. Resource group name Multiple existing security groups on AWS When creating a cluster on AWS, you can select multiple existing security groups. This option is available only when an existing VPC is selected. Create a cluster on AWS EBS volume encryption on AWS You can optionally configure encryption for EBS volumes attached to cluster instances running on EC2. Default or customer-managed encryption keys can be used. EBS encryption on AWS Shared VPCs on GCP When creating a cluster on Google Cloud, you can place it in an existing shared VPC. Shared networks on GCP GCP volume encryption By default, Google Compute Engine encrypts data at rest stored on disks. You can optionally configure encryption for the encryption keys used for disk encryption. Customer-supplied (CSEK) or customer-managed (CMEK) encryption keys can be used. Disk encryption on GCP Workspaces Cloudbreak introduces a new authorization model, which allows resource sharing via workspaces. In addition to a default personal workspace, users can create additional shared workspaces. Workspaces Operations audit logging Cloudbreak records an audit trail of the actions performed by Cloudbreak users as well as those performed by the Cloudbreak application. Operations audit logging Updating long-running clusters Cloudbreak supports updating base image's operating system and any third party packages that have been installed, as well as upgrading Ambari, HDP and HDF. Updating OS and tools on long-running clusters and Updating Ambari and HDP/HDF on long-running clusters HDP 3.1 Cloudbreak introduces two default HDP 3.1 blueprints and allows you to create your custom HDP 3.1 blueprints. Default cluster configurations HDF 3.3 Cloudbreak introduces two default HDF 3.3 blueprints and allows you to create your custom HDP 3.3 blueprints. To get started, refer to How to create a NiFi cluster HCC post. Default cluster configurations Recipe parameters Supported parameters can be specified in recipes as variables by using mustache kind of templating with "{{{ }}}" syntax. Writing recipes and Recipe parameters Shebang in Python recipes Cloudbreak supports using shebang in Python scripts run as recipes. Writing recipes Technical preview features The following features are technical preview (not suitable for production): Feature Description Documentation AWS GovCloud (TP) You can install Cloudbreak and create Cloudbreak-managed clusters on AWS GovCloud. Deploying on AWS GovCloud Azure ADLS Gen2 (TP) When creating a cluster on Azure, you can optionally configure access to ADLS Gen2. This feature is technical preview. Configuring access to ADLS Gen2 New and changed data lake blueprints (TP) Cloudbreak includes three data lake blueprints, two for HDP 2.6 (HA and Atlas) and one for HDP 3.1. Note that Hive Metastore has been removed from the HDP 3.x data lake blueprints, but setting up an external database allows all clusters attached to a data lake to connect to the same Hive Metastore. To get started with data lakes, refer to How to create a data lake with Cloudbreak 2.9 HCC post. Working with data lakes Default blueprints Cloudbreak 2.9.0 includes the following HDP 2.6, HDP 3.1, and HDF 3.3 workload cluster blueprints. In addition, HDP 3.1 and HDP 2.6 data lake blueprints are available as technical preview. Note that Hive Metastore has been removed from the HDP 3.x data lake blueprints, but setting up an external database allows all clusters attached to a data lake to connect to the same Hive Metastore. Documentation links How to create a data lake with Cloudbreak 2.9 (HCC post) How to create a NiFi cluster (HCC post) Cloudbreak 2.9.0 documentation (Official docs) Release notes (Official docs) ... View more

Cloudbreak 2.7.2 maintenance release is now available. If you are using an earlier version of Cloudbreak, you can upgrade now to pick up the latest bug fixes. If you are new to Cloudbreak, you can get started by launching Cloudbreak on AWS, Azure, GCP, or OpenStack from a template. Useful links: Release notes Upgrade steps Get started ... View more

Cloudbreak 2.8.0 Technical Preview release is now available! New features Cloudbreak 2.8.0 TP introduces the following new features: Feature Description Documentation AWS GovCloud Cloudbreak supports installing Cloudbreak and creating Cloudbreak-managed clusters on AWS GovCloud. Deploying on AWS vs AWS GovCloud EBS volume encryption on AWS You can optionally configure encryption for EBS volumes attached to cluster instances running on EC2. Default or customer-managed encryption keys can be used. EBS encryption on AWS GCP volume encryption By default, Compute Engine encrypts data at rest stored on disks. You can optionally configure encryption for the encryption keys used for disk encryption. Customer-supplied (CSEK) or customer-managed (CMEK) encryption keys can be used. Disk encryption on GCP User authorization Cloudbreak introduces a new authorization model, which allows resource sharing via organizations. User authorization Operations audit logging Cloudbreak records an audit trail of the actions performed by Cloudbreak users as well as those performed by the Cloudbreak application. Operations audit logging Updating long-running cluster Cloudbreak supports updating base image's operating system and any third party packages that have been installed. Updating long-running clusters Data lake HA and Atlas support Cloudbreak includes two data lake blueprints: Data lake HA blueprint Data lake blueprint including Atlas (HA is not supported) Working with Data Lakes (TP) Multiple existing security groups on AWS Multiple existing security groups can be specified when creating a cluster via CLI on AWS. Multiple existing security groups on AWS Shebang in Python recipes Cloudbreak supports using shebang in Python scripts run as recipes. Writing recipes HDF 3.2 Cloudbreak can be used to deploy HDF 3.2 clusters by using one of the two default HDF 3.2 blueprints: Flow Management clusters with Apache NiFi Messaging clusters with Apache Kafka. Default cluster configurations For more information on what changed in Cloudbreak 2.8.0 TP, refer to Release Notes. Redesigned documentation Starting with Cloudbreak 2.8.0, Cloudbreak documentation is published in the same format (HTML and PDF) as other Hortonworks documentation. You can access the available publications here: https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.8.0/index.html Here is what you should expect to find in each publication: RELEASE NOTES Release Notes: New features, behavioral changes, known issues, fixed issues, and image catalog updates. CONCEPTS Introduction to Cloudbreak: High-level conceptual information about Cloudbreak. Cloudbreak Security Overview: High-level conceptual information related to security in Cloudbreak. INSTALLATION & UPGRADE Cloudbreak Deployment Options: Introduction to Cloudbreak and cluster installation options, helping you decide which option to use. Quickstart on AWS/Azure/GCP: Instructions for how to install Cloudbreak from a template. This is typically not suitable for production. Installing Cloudbreak on AWS/Azure/GCP/OpenStack: Instructions for how to install Cloudbreak on your own VM. This is typically used for production. Upgrading Cloudbreak: Instructions for how to upgrade your Cloudbreak instance. Always refer to upgrade documentation for the Cloudbreak version that you are upgrading to. Installing Cloudbreak CLI: Instructions for installing and configuring Cloudbreak CLI. HOW TO Creating a Cloudbreak Credential on AWS/Azure/GCP/OpenStack: Cloud provider-specific instructions for creating a Cloudbreak credential. Creating a Cluster on AWS/Azure/GCP/OpenStack: Cloud provider-specific instructions for creating HDP and HDF clusters and an overview of advanced cluster options. Accessing Clusters: Information about user accounts and instructions for accessing Cloudbreak-managed clusters. Managing Clusters: Instructions for how to perform cluster management tasks such as resizing, stopping, configuring autoscaling, configuring an SSL certificate, and updating OS on cluster VMs. Advanced Cluster Options: Instructions for how to configure advanced cluster options such as custom images, recipes, Kerberos security, disk encryption, external databases, LDAP/AD, and more. Configuring Access to Cloud Data: Instructions for how to configure access from a cluster created via Cloudbreak to Amazon S3, ADLS, WASB, or GCS. Working with Data Lakes: Introduction to data lakes and data lake setup-steps. Advanced Cloudbreak Configuration: Instructions on how to set up an external Cloudbreak database, LDAP/AD or Cloudbreak, and other advanced Cloudbreak configurations. Managing and Monitoring Cloudbreak: (NEW) Information about Cloudbreak’s user authentication model and operational audit logging. TROUBLESHOOTING Troubleshooting Cloudbreak: Steps for troubleshooting Cloudbreak. Also includes the location of Cloudbreak logs. REFERENCE CLI Reference: Cloudbreak CLI reference, including command examples. Developer Documentation: Links to API and other developer docs.' Get started with Cloudbreak Use the following links to install or upgrade Cloudbreak: Quickstart on AWS/Azure/GCP Upgrading Cloudbreak Cloudbreak Deployment Options ... View more

Although as of Cloudbreak 2.8.0 there is no way to manage cluster templates in the Cloudbreak web UI, you can generate a reusable template from the UI and then reuse it for creating clusters via Cloudbreak CLI. In this blog post, I describe the steps for how to do this. Step 1: In the web UI, provide cluster parameters and generate a cluster template 1.In the Cloudbreak web UI, navigate to Clusters and then click on Create cluster. 2.To obtain a full list of cluster options, click on Advanced. 3.Provide all the parameters for your cluster. 4.Once done, on the last page of the create cluster wizard, click on Show CLI Command. 5.Click on Copy the JSON to copy the generated CLI skeleton. 6.Paste it into a text editor and save the file onto your computer in .JSON format. 7.Open the file and review it to ensure that no parameters are missing. Save the file once you are done editing. Step 2: Create clusters via CLI from the template 1.Download and configure Cloudbreak CLI. Refer to Install CLI and Configure the CLI. 2.To create a cluster from the template generated in the previous step, use the following command: cb cluster create --cli-input-json <path-to-the-cluster-template> For example: cb cluster create --cli-input-json /Users/myuser/Documents/mytemplate.json More options If you would like to create multiple clusters from one template, you will have to provide a different name for each cluster. You can do this by not specifying the cluster name in the template, but passing it with each cb cluster create command. For example: cb cluster create --name mycluster1 --cli-input-json /Users/myuser/Documents/mytemplate.json As an alternative for generating cluster template from the UI, you can use the following command to generate a cluster template: cb cluster generate-template However, when specifying many advanced options (such as custom Ambari and stack repos), I find it easier, faster, and more precise to generate a template from the UI. ... View more

In the following video, I demonstrate how to create a Cloudbreak credential on Google Cloud Video link: https://youtu.be/uVYpgz9m4eE The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the roles that need to be assigned to the service account, refer to Service account for GCP credential documentation. If you are using a corporate Google Cloud account you may be unable to perform some of the steps (such as service account creation and role assignment) by yourself and you may have to contact your Google Cloud admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more

In the following video, I demonstrate how to create a key-based Cloudbreak credential on AWS Video link: https://youtu.be/SIYhzlMQf8Y The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the policy that need to be assigned to the user, refer to Create CredentialRole documentation. If you are using a corporate AWS account you may be unable to perform some of the steps (such as user creation and policy assignment) by yourself and you may have to contact your AWS admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more

In the following video, I demonstrate how to create an app-based Cloudbreak credential on Azure Video link: https://youtu.be/NYQZd2AmM6g The Cloudbreak version used in this video is Cloudbreak 2.7.1. If you are using a corporate Azure account you may be unable to perform some of the steps (such as role creation and policy assignment) by yourself and you may have to contact your Azure admin to perform these steps for you. If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com. ... View more