Error: Failed to verify the credential: User: arn:aws:iam::<roleid>:user/assume-only-user is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<roleid>:role/<rolename> Reason: The Cloubdreak instance is not authorized to assume the role that you are trying to register for the Cloudbreak credential. How to fix: If you are using hosted Cloudbreak, you may be entering an incorrect AWS account ID when creating the role for Cloudbreak credential. Contact the Cloudbreak team to confirm the AWS account ID. ... View more

Due to a recent change in Google Cloud's handling of requests, all clusters created on Google Cloud Platform with Cloudbreak Cloudbreak 2.4.0 (or earlier 2.x) and 1.16.5 (or earlier 1.x) fail with an error similar to: Infrastructure creation failed. Reason: com.sequenceiq.cloudbreak.cloud.gcp.GcpResourceException: Error during status check: [ resourceType: GCP_DISK, resourceName: hdf-test, stackId: 5, operation: hdftest-s-2-20180323123311 ] [ Cause message: 403 Forbidden { "code" : 403, "errors" : [ { "domain" : "global", "message" : "Required 'compute.globalOperations.get' permission for 'projects/*******/global/operations/operation-1521808391647-56813a099af18-300f33eb-8a28c20c'", "reason" : "forbidden" } ], "message" : "Required 'compute.globalOperations.get' permission for 'projects/*******/global/operations/operation-1521808391647-56813a099af18-300f33eb-8a28c20c'" } ] In order to address the issue, we created a 2.4.1 maintenance release. You can upgrade to Cloudbreak 2.4.1 by using cbd update. For detailed update steps, refer to Cloudbreak 2.4.1 docs. > As a side note, Cloudbreak 2.4.1 also includes a fix for time-based scaling. If you are using Cloudbreak 1.x and do not want to update to Cloudbreak 2.x, you can update to Cloudbreak 1.16.6 by following these steps: 1.Download the CBD binary: curl -Ls public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_1.16.6_$(uname)_x86_64.tgz | sudo tar -xz -C /bin cbd 2.After updating, restart Cloudbreak (must be done from the deployment directory): cbd restart ... View more

Cloudbreak 2.5.0 Technical Preview is available now. Here are the highlights: Creating HDF Clusters You can use Cloudbreak to create HDF clusters from base images on AWS, Azure, Google Cloud and OpenStack. In the Cloudbreak web UI, you can do this by selecting "HDF 3.1" under Platform Version and then selecting an HDF blueprint. Cloudbreak includes one default HDF blueprint "Flow Management: Apache NiFi" and supports uploading your own custom HDF 3.1.1 NiFi blueprints. Note the following when creating NiFi clusters: When creating a cluster, open 9091 TCP port on the NiFi host group. Without it, you will be unable to access the UI. Enabling kerberos is mandatory. You can either use your own kerberos or select for Cloudbreak to create a test KDC. Although Cloudbreak includes cluster scaling (including autoscaling), scaling is not fully supported by NiFi. Downscaling NiFi clusters is not supported - as it can result in data loss when a node is removed that has not yet processed all the data on that node. There is also a known issue related to scaling listed in the Known Issues below. For updated create cluster instructions, refer to Creating a Cluster instructions for your chosen cloud provider. For updated blueprint information, refer to Default Blueprints. For a tutorial on creating a NiFi cluster with Cloudbreak, refer to the following HCC post. > HDF options in the create cluster wizard: Using External Databases for Cluster Services You can register an existing external RDBMS in the Cloudbreak UI or CLI so that it can be used for those cluster components which have support for it. After the RDBMS has been registered with Cloudbreak, it will be available during the cluster create and can be reused with multiple clusters. Only Postgres is supported at this time. Refer to component-specific documentation for information on which version of Postgres (if any) is supported. For more information, refer to Register an External Database. > UI for registering an external DB: > UI for selecting a previously registered DB to be attached to a specific cluster: Using External Authentication Sources (LDAP/AD) for Clusters You can configure an existing LDAP/AD authentication source in the Cloudbreak UI or CLI so that it can later be associated with one or more Cloudbreak-managed clusters. After the authentication source has been registered with Cloudbreak, it will be available during the cluster create and can be reused with multiple clusters. For more information, refer to Register an Authentication Source. > UI for registering an existing LDAP/AD with Cloudbreak: > UI for selecting a previously registered authentication source to be attached to a specific cluster: Modifying Existing Cloudbreak Credentials Cloudbreak allows you to modify existing credentials by using the edit option available in Cloudbreak UI or by using the credential modify command in the CLI. For more information, refer to Modify an Existing Credential. > UI for managing Cloudbreak credentials: Configuring Cloudbreak to Use Existing LDAP/AD You can configure Cloudbreak to use your existing LDAP/AD so that you can authenticate Cloudbreak users against an existing LDAP/AD server. For more information, refer to Configuring Cloudbreak for LDAP/AD Authentication. Launching Cloudbreak in Environments with Restricted Internet Access or Required Use of Proxy You can launch Cloudbreak in environments with limited or restricted internet access and/or required use of a proxy to obtain internet access. For more information, refer to Configure Outbound Internet Access and Proxy. Auto-import of HDP/HDF Images on OpenStack When using Cloudbreak on OpenStack, you no longer need to import HDP and HDF images manually, because during your first attempt to create a cluster, Cloudbreak automatically imports HDP and HDF images to your OpenStack. Only Cloudbreak image must be imported manually. More To learn more, check out the Cloudbreak 2.5.0 docs. To get started with HDF, check out this HCC post. If you are part of Hortonworks, you can check out the new features by accessing the internal hosted Cloudbreak instance. ... View more

In this tutorial, I create a NiFi cluster from the default blueprint provided with Cloudbreak 2.5.0 TP, but this post has been updated to reflect what is available in Cloudbreak 2.9.0. Update for Cloudbreak 2.9.0 This post was originally written for Cloudbreak 2.5.0 TP, which introduced support for creating HDF Flow Management (NiFi) clusters. It has been updated to reflect the latest features available in Cloudbreak 2.9.0 general availability release. Cloudbreak 2.5.0 TP introduced support for creating HDF Flow Management (NiFi) clusters.The subsequent release, Cloudbreak 2.6.0 TP, introduced support for creating HDF Messaging Management (Kafka) clusters. Cloudbreak 2.7.0 GA introduced these as general availability features. In Cloudbreak 2.9.0 GA, two HDF 3.3 blueprints are included by default, one for Flow Management (NiFi) and one for Messaging Management (Kafka). What is Cloudbreak? Cloudbreak simplifies the provisioning, management, and monitoring of on-demand HDP and HDF clusters in virtual and cloud environments. It leverages cloud infrastructure to create host instances, and uses Apache Ambari via Ambari blueprints to provision and manage HDP and HDF clusters. Cloudbreak allows you to create HDP and HDF clusters using the Cloudbreak web UI, Cloudbreak CLI, and Cloudbreak REST API. Clusters can be launched on public cloud infrastructure platforms Microsoft Azure, Amazon Web Services (AWS), and Google Cloud Platform, and on the private cloud infrastructure platform OpenStack. Support for creating NiFi clusters was introduced in Cloudbreak 2.5.0 Technical Preview and was GA'd in Cloudbreak 2.7.0. Prerequisites Cloud Provider Account In order to use Cloudbreak, you must have access to a cloud provider account (AWS, Azure, Google Cloud, OpenStack) on which resources can be provisioned. Launch Cloudbreak If you part of Hortonworks, you have access to the hosted Cloudbreak instance. If you do not have access to this instance, you must launch Cloudbreak on your chosen cloud platform. The instructions for launching Cloudbreak are available here: Cloudbreak Deployment Options Quickstart on AWS/Azure/GCP Installing Cloudbreak on AWS/Azure/GCP/OpenStack Create a Cloudbreak Credential Once your Cloudbreak instance is running, you must create a Cloudbreak credential before you can start creating HDP or HDF clusters. Why do you need this? By creating a Cloudbreak credential, you provide Cloudbreak with the means to authenticate with your cloud provider account and provision resources (virtual networks, VMs, and so on) on that account. Creating a Cloudbreak credential is always required, even if Cloudbreak instance is running on your cloud provider account. The instructions for creating a Cloudbreak credential are available here: Creating a Cloudbreak Credential on AWS/Azure/GCP/OpenStack > Tip: When using a corporate cloud provider account, you are unlikely to be able to perform all the required prerequisite steps by yourself and you may therefore need to contact your IT so that they can perform some of the steps for you. For related tips, refer to this HCC post. Create a Flow Management Cluster Creating clusters is possible from Cloudbreak web UI and Cloudbreak CLI. It’s best to get started with the UI before attempting to use the CLI. 1.Log in to the Cloudbreak UI. 2.Click Create Cluster and the Create Cluster wizard is displayed. By default, Basic view is displayed. You can click Advanced to see more options, but in this post we are just addressing basic parameters. 3.On the General Configuration page, specify the following general parameters for your cluster: Select Credential: Select the credential that you created as a prerequisite. After you have selected the credential, the wizard options will be adjusted for the cloud platform that the credential can be used for and parameters such as regions will be populated. As you can see in my screenshot, I selected an AWS-specific credential and my cluster will be created on AWS. Cluster Name: Enter some name for your cluster, for example, “nifi-test”. Region: Select the region that you would like to use (typically your local region). On some cloud providers, you can also select an availability zone within a region. Platform Version: Select “HDF 3.3” (Sorry for the old screenshot showing 3.2). Cluster Type: Select “Flow Management: Apache NiFi”. This is a default blueprint which includes Apache NiFi. 4.When done, click Next. 5.On the Hardware and Storage page, Cloudbreak pre-populates recommended instance types/count and storage type/size. You may adjust these depending on how many nodes and storage you want and of what type. By default, a 2-node cluster will be created with one node in each host group (Services host group and NiFi host group). 6.Before proceeding to the next page, you must select the host group on which Ambari Server will be install. Under “Services” host group, check “Ambari Server” so that Ambari Server is installed on that host group: 7.When done, click Next. 8.On the Network and Availability page, you can proceed with the default settings or adjust the settings in the following way: Select Network: If you do not make a selection, a new network will be created on the cloud provider. If you already have a network that you would like to use, you can select it here; otherwise you can keep the default, just note - if you are using a shared account - that there are limits to how many virtual networks can be created per region within a given account. Select Subnet: If you do not make a selection, a new subnet will be created on the cloud provider within the network selected under “Select Network”. Subnet (CIDR): By default, 10.0.0.0/16 is used. 9. On the Gateway Configuration page, do not change anything. Just click Next. 10. On the Network Security Groups page, On NiFi host group, specify a TCP rule to open port 9091 to your public IP and click + to add it. On the Services host group, specify a TCP rule to open 61443. This port is used by NiFi Registry. Your configuration should look like this: > Tip: If you are planning to use NiFi processors that require additional ports, add additional rules to open these ports. > Tip: By default, Cloudbreak creates a new security group for each host group. By default, ports 9443, 22, and 443 are open to all (0.0.0.0/0) on the Services host group (because this is where Ambari Server is installed); and port 22 is open to all (0.0.0.0/0) on the NiFi host group. These settings are not suitable for production. If you are planning to leave your cluster running for longer than a few hours, review the guidelines documented here and limit the access by (1) deleting the default rules and (2) adding new rules by setting the CIDR to “My IP” and “Custom” (use “Custom” for specifying the Cloudbreak instance IP). 9.On the Security page, provide the following information: Cluster User: Enter the name for the cluster user. Default is “admin”. You will use this to log in to Ambari web UI. Password and Confirm Password: Enter the password for the cluster user. You will use this to log in to Ambari web UI. SSH public key: Paste your existing SSH public key or select an SSH public key that you have previously uploaded to your cloud provider account. Later, in order to access the cluster nodes via SSH, you will have to provide the matching private key. Enable Kerberos Security: If you are just getting started, select “Use Test KDC” to have a new test KDC provisioned for the cluster. > Warning: Make sure not to disable Kerberos. If you don’t have one, select to create a test KDC. If you use the default Flow Management blueprint without enabling Kerberos, the NiFi UI will be inaccessible unless you configure an SSL certificate OR you register and use an existing LDAP. 10.At this point, you have provided all parameters required to create your cluster. Click CREATE CLUSTER to start cluster creation process. 11.You will be redirected to the cluster dashboard and the cluster status presented on the corresponding tile will be “Create in progress” (blue color). When the cluster is ready, its status will change to “Running”: Access and Manage Clusters Once the status of your cluster changes to “Running”, click on the cluster tile to view cluster details where you can find information related to your cluster and access cluster-related options. Note the following options: 1.Click on the link under Ambari URL to access Ambari web UI in the browser: 2.Log in to the Ambari web UI by using the cluster user and password created when creating a cluster. Since Ambari web UI is set up with a self-signed SSL certificate, the first time you access it your browser will warn you about an untrusted connection and will ask you to confirm a security exception. Once you have logged in, you can access NiFi service from the Ambari dashboard: Nifi UI link is available from Quick Links: 3.To access cluster nodes via SSH, use: The “cloudbreak” user The private key corresponding to the public key that you provided/selected when creating a cluster Obtain the VM public IP address from the Hardware pane in the cluster details: For example, on Max OS X: ssh -i "mytest-kp.pem" cloudbreak@52.25.169.132 4.Cloudbreak web UI provides the options to Stop/Start, and Sync the cluster with the cloud provider. Once you don’t need the cluster, you can terminate it by using the Terminate option available in the cluster details. > Resizing and autoscaling: In general, downscaling NiFi clusters is not supported - as it can result in data loss when a node is removed that has not yet processed all the data on that node. Upscaling is supported, but there is a known issue which requires you to manually update the newly added hosts (see Known Issues). 5.The Show CLI command option allows you to generate a JSON template for the existing cluster; the template can later be used to automate cluster creation with Cloudbreak CLI. 6.You can download Cloudbreak CLI by selecting Download CLI from the navigation pane. The CLI is available for Mac OS X/Windows/Linux. 7.You only need to configure the CLI once so that it can be used with your Cloudbreak instance: ./cb configure --server<cloudbreak IP> --username <cloudbreak-user> --password <cloudbreak-password> 8.Once it has been configured, you can view available commands by using: ./cb Advanced Cluster Options Cloudbreak includes additional advanced options, some of which are cloud platform-specific. To review them, refer to the following docs: Creating a Cluster on AWS/Azure/GCP/OpenStack Learn More Cloudbreak 2.9.0 docs Creating HDF clusters ... View more

This article discusses prerequisites required for Cloudbreak credential and addresses common issues. This article applies to Cloudbreak 2.4.0. When getting started with Cloudbreak, the most common pain point is creating the Cloudbreak credential, or - more precisely - meeting the prerequisites for the Cloudbreak credential. Why do you even need this Cloudbreak credential? Cloudbreak not only runs on your cloud provider account, but once you start using, it provisions resources (such as VMs) that run as part of your account and that you or your organization must pay for. Cloudbreak credential provides the means for Cloudbreak to authenticate with your cloud provider account and provision such resources. The exact details vary depending on the cloud provider: On Azure this is typically done by creating an app registration in Azure Active Directory and providing its details to Cloudbreak. On AWS this typically happens by assigning a proper IAM role to Cloudbreak. Cloudbreak can assume this role in order to create resources on your behalf. On Google Cloud this is typically done by creating a Service Account that Cloudbreak can use to create resources. Azure Credential There are two ways in which Cloudbreak can access your Azure account: interactive and app-based. Interactive The first way is via the interactive credential, where Cloudbreak automated the app and service principal creation and role assignment, so the only input that you need to provide is the Subscription ID and Directory ID. You can definitely use this if you are evaluating Cloudbreak by deploying it on your own Azure account. However, if you are using a corporate account, you will most likely be unable to use the interactive credential at all, because your account must have the "Owner" role (or its equivalent) in order for Cloudbreak to perform the interactive credential creation steps. The error that you will get will be: You don't have enough permissions to assign roles, please contact your administrator Also - just to throw it up there - when using this method, you must be able log in to your Azure account. > This is true for Cloudbreak 2.4.0, but may change in future releases. App-based If you do not meet the requirements for using the interactive credential, you are left with the second option, which is the app-based credential. The advantage of the app-based credential creation is that it allows you to create a credential without logging in to the Azure account, as long as you have been given all the required information. You must provide your Subscription ID and Directory ID (called “Tenant ID” in Cloudbreak UI), you must provide information for your previously created Azure AD application (its ID and key which allows access to it). One tip here is that after registering an Azure application you may have to ask your Azure administrator to perform the step of assigning the "Contributor" role to it: Your account will most likely not have the required permissions and you will get an error. Once your Azure admin performed the role assignment, you can perform the create credential task in Cloudbreak. Related docs: Create Cloudbreak Credential on Azure AWS Credential On AWS, there are two ways for Cloudbreak to authenticate with your AWS account and create resources on your behalf: key-based credential and role-based credential. Key-based To use this, you must be able to provide your AWS access key and secret key pair. If you don’t have these, you may be able to generate a new access and secret key pair from the IAM Console > Users. If you are able to generate these, then this is the easiest way to get started. However, when using a corporate account, you may want to or you may have to use the more complicated role-based credential. Role-based Role-based credential utilizes IAM (Identity Access Management) roles. There are two IAM roles involved: When launching Cloudbreak, you are required to provide an IAM role with AssumeRole policy assigned. The AssumeRole allows Cloudbreak to use the CredentialRole described in the next bullet. After Cloudbreak is running, when you create a Cloudbreak credential, you must provide an IAM Role ARN for another IAM role (which is called CredentialRole in the documentation). This role provides a set of permissions required for provisioning resources. If you chose (or were forced to choose) this approach, you may need to contact your AWS admin to help you out, as you may not be able to create IAM roles. Another tip is that you should make sure to include all the permissions mentioned in the CredentialRole policy definition. If you miss some of them, you will get an access-related error when creating a cluster. Reference: AssumeRole policy definition for Cloudbreak: { "Version": "2012-10-17", "Statement": { "Sid": "Stmt1400068149000", "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": "*" } } Policy definition required for Cloudbreak credential: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudformation:CreateStack", "cloudformation:DeleteStack", "cloudformation:DescribeStackEvents", "cloudformation:DescribeStackResource", "cloudformation:DescribeStacks" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "ec2:AllocateAddress", "ec2:AssociateAddress", "ec2:AssociateRouteTable", "ec2:AuthorizeSecurityGroupIngress", "ec2:DescribeRegions", "ec2:DescribeAvailabilityZones", "ec2:CreateRoute", "ec2:CreateRouteTable", "ec2:CreateSecurityGroup", "ec2:CreateSubnet", "ec2:CreateTags", "ec2:CreateVpc", "ec2:ModifyVpcAttribute", "ec2:DeleteSubnet", "ec2:CreateInternetGateway", "ec2:CreateKeyPair", "ec2:DisassociateAddress", "ec2:DisassociateRouteTable", "ec2:ModifySubnetAttribute", "ec2:ReleaseAddress", "ec2:DescribeAddresses", "ec2:DescribeImages", "ec2:DescribeInstanceStatus", "ec2:DescribeInstances", "ec2:DescribeInternetGateways", "ec2:DescribeKeyPairs", "ec2:DescribeRouteTables", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcs", "ec2:DescribeSpotInstanceRequests", "ec2:DescribeVpcAttribute", "ec2:ImportKeyPair", "ec2:AttachInternetGateway", "ec2:DeleteVpc", "ec2:DeleteSecurityGroup", "ec2:DeleteRouteTable", "ec2:DeleteInternetGateway", "ec2:DeleteRouteTable", "ec2:DeleteRoute", "ec2:DetachInternetGateway", "ec2:RunInstances", "ec2:StartInstances", "ec2:StopInstances", "ec2:TerminateInstances" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:ListRolePolicies", "iam:GetRolePolicy", "iam:ListAttachedRolePolicies", "iam:ListInstanceProfiles", "iam:PutRolePolicy", "iam:PassRole", "iam:GetRole" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "autoscaling:CreateAutoScalingGroup", "autoscaling:CreateLaunchConfiguration", "autoscaling:DeleteAutoScalingGroup", "autoscaling:DeleteLaunchConfiguration", "autoscaling:DescribeAutoScalingGroups", "autoscaling:DescribeLaunchConfigurations", "autoscaling:DescribeScalingActivities", "autoscaling:DetachInstances", "autoscaling:ResumeProcesses", "autoscaling:SuspendProcesses", "autoscaling:UpdateAutoScalingGroup" ], "Resource": [ "*" ] } ] } Related docs: IAM Roles (AWS) Authentication with AWS Create Cloudbreak Credential Google Cloud Credential In order to launch clusters on GCP via Cloudbreak, you must have a Service Account that Cloudbreak can use to create resources. In addition, you must also have a P12 key associated with that account. Usually, a user with an "Owner" role can assign roles to new and existing service accounts from IAM & Admin > IAM. If you are using your own account, you should be able to perform this step, but if you are using a corporate account, you will likely have to contact your Google Cloud admin. The service account must have the following roles enabled: Compute Engine > Compute Image User Compute Engine > Compute Instance Admin (v1) Compute Engine > Compute Network Admin Compute Engine > Compute Security Admin Storage > Storage Admin Note that you must include these exact permissions or else you will run into problems. Related docs: Service Accounts (Google Cloud) Service Account for Cloudbreak Create Cloudbreak Credential General Tips 1. Although the UI option and CLI command for creating a Cloudbreak credential appear to be simple, the related prerequisites are actually complex and, if you are using a corporate cloud account, you may have to contact your cloud account administrator to perform some of the prerequisite steps for you. 2. When meeting the prerequisites for Cloudbreak credential (for example trying to create an IAM role on AWS, trying to create an app registration on Azure, trying to create a Service Account on Google Cloud), an error related to permissions/authorization/roles or an inability to access certain options may mean that you do not have the permissions to perform certain actions and you may have to ask your cloud admin for help. 3. Once the credential has been created and you are trying to create a cluster, an error related to permissions/authorization/roles usually means that the scope of permissions assigned to Cloudbreak (via IAM role on AWS, Contributor role equivalent on Azure, Service Account on Google Cloud) is insufficient. When the scope of permissions is insufficient, you may experience other issues, such as being unable to see provider regions and instance types in the create cluster UI wizard or CLI. As far as I know, Cloudbreak 2.4.0 does not check the permissions provided in the policy assigned to the IAM role (on AWS) or Service Account (Google Cloud). On Azure, on the other hand, these checks are in place. ... View more

Cloudbreak 2.4.0 is now live! What is Cloudbreak: Cloudbreak simplifies cluster provisioning on public cloud infrastructure platforms Microsoft Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP), and on the private cloud infrastructure platform OpenStack. Here are the highlights of what’s new in Cloudbreak 2.4.0 compared with Cloudbreak 1.16.x: New UI/UX Cloudbreak 2.4.0 introduces a new user interface. All major options are now easily accessible from the collapsible navigation menu. All UI options and wizards have been redesigned in order to make cluster creation and management more intuitive. The following screenshot shows the cluster dashboard: The following screenshot shows cluster details page: > Note that autoscaling is available from the Autoscaling tab. > Note the Show CLI Command available from Actions menu. This allows you to generate a CLI template of the cluster. A similar option is available on the last page of the create cluster wizard. The following screenshot shows the create cluster wizard: > Note the Basic/Advanced toggle button, which allows you to switch between the basic and advanced wizard view. In the Settings you can select which view you would like to see by default. New CLI Cloudbreak 2.4.0 introduces the new CLI tool, which replaces Cloudbreak Shell. All commands start with a singular object followed by an action, for example, blueprint create and blueprint list. To download the CLI, select Download CLI from the navigation pane. The following commands are available: blueprint blueprint related operations cloud information about cloud provider resources cluster cluster related operations configure configure the server address and credentials used to communicate with this server credential credential related operations imagecatalog imagecatalog related operations recipe recipe related operations help, h Shows a list of commands or help for one command Refer to Install CLI, Get Started with the CLI, and CLI Reference. Support for Kerberos Creating Kerberos-enabled clusters is supported: Use an existing MIT KDC or Active Directory for production deployments. Use a test KDC for evaluation purposes. Refer to Enabling Kerberos Security. Configuring an External Database for Cloudbreak Using an external RDBMS for Cloudbreak is supported and recommended for production environments. For configuration instructions, refer to Configuring External Cloudbreak Database. Migrating Cloudbreak Instance Migrating Cloudbreak from one machine to another is supported. For migration instructions, refer to Moving a Cloudbreak Instance. Prewarmed Images To accelerate cluster creation, CLoudbreak 2.4.0 introduces prewarmed images, which include the operating system, as well as the default version of Ambari and HDP. By default, Cloudbreak 2.4 launches clusters from these prewarmed images, instead of using base images (which were used by default in earlier versions of Cloudbreak). Default base images are still available in case you would like to use different Ambari and HDP versions than those provided with prewarmed images. For more information, refer to Prewarmed and Base Images. Providing Your Own SDK Providing your own JDK on a custom base image is supported. For instructions, refer to "Advanced topics" in the https://github.com/hortonworks/cloudbreak-images repository. Generating CLI Templates After specifying the parameters for your cluster in the Cloudbreak web UI, you can copy the content of the CLI JSON file that can be used to create a cluster via Cloudbreak CLI. For more information, refer to Obtain Cluster JSON Template from the UI. Furthermore, Cloudbreak web UI includes an option in the UI which allows you to generate the createcommand for resources such as credentials, blueprints, clusters, and recipes. For more information, refer to Obtain CLI Command from the UI. New Recipe Types New types of recipes are introduced: PRE-AMBARI-START (new, useful for configuring Ambari prior to start) POST-AMBARI-START (formerly known as PRE) POST-CLUSTER-INSTALL (formerly known as POST) PRE-TERMINATION (new, useful for cluster cleanup pre-termination tasks) Refer to Recipes documentation. Disabling Cloud Providers You can hide cloud providers available in Cloudbreak by adding the CB_ENABLEDPLATFORMS environment variable in Profile and setting it to the provider(s) that you would like to have available. For more information, refer to Disable Providers. Support for Ambari 2.6.1 (Specifically 2.6.1.3+) Cloudbreak 2.4.0 introduces support for Ambari 2.6 and uses Ambari 2.6.1.3 by default. If you would like to use Ambari 2.6, you must use Ambari version 2.6.1.3 or newer. If you decide to use Ambari 2.6.1, you should be aware of the following: Ambari 2.6.1 or newer does not install the mysqlconnector; therefore, when creating a blueprint for Ambari 2.6.1 or newer do not include the MYSQL_SERVER component for Hive Metastore in your blueprint. Instead, you have two options described in Creating a Blueprint. If you would like to use Oozie, you must manually install Ext JS. The steps are described in Cannot Access Oozie Web UI. To enable LZO compression in your HDP cluster, you must check the "Enable Ambari Server to download and install GPL Licensed LZO packages?" during cluster creation. The option is available under Security > Prewarmed and Base Images. Ambari Master Key Cloudbreak 2.4.0 allows you to specify the Ambari Master Key. The Ambari Server Master Key is used to configure Ambari to encrypt database and Kerberos credentials that are retained by Ambari as part of the Ambari setup. The option is available on the Security page of the create cluster wizard. Get Started with Cloudbreak 2.4.0 GA To get started on your chosen platform, refer to Cloudbreak 2.4.0 documentation. > Note that Cloudbreak 2.4.0 documentation was written from scratch, rather than being based on Cloudbreak 1.16.x documentation. The steps for meeting the prerequisites, launching Cloudbreak, and creating a Cloudbreak credentials are described in detail, including numerous screenshots. They are appropriate for users who want to get started with Cloudbreak but are not familiar with the AWS, Azure, and Google Cloud cloud providers. Get started now with Cloudbreak 2.4.0! ... View more