Cloudbreak 2.4.2 has been released. In addition to a few bug fixes, the release comes with new base and prewarmed images including Ambari 2.6.2 and HDP 2.6.5 (Cloudbreak 2.4.0 and 2.4.1 came with earlier Ambari and HDP versions). For documentation, see https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.2/content/index.html For release notes, see https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.2/content/releasenotes/index.html ... View more

Cloudbreak 2.4.0 and newer includes a neat feature which allows you to obtain CLI commands from the web UI. This feature can save you a lot of time if you are just getting started with the CLI. Prerequisites You must have an instance of Cloudbreak running. You downloaded and configured Cloudbreak CLI. if you still need to do this, refer to Installing the CLI. Obtain Cluster JSON from the UI By far the most useful feature is being able to obtain cluster JSON file from the web UI. There are two easy ways to do this. Option 1 (Does not require creating a cluster) Navigate to the create cluster wizard and specify all required parameters. Once done, on the last page of the wizard there is an option to SHOW CLI COMMAND, which allows you to generate a JSON skeleton of your cluster: You can click COPY the JSON, paste it to a text edit, and save it as template.json. Next, you can copy the command listed in the web UI below the CLI skeleton to create a cluster from the template.json file: cb cluster create --cli-input-json template.json --name cli-cluster Option 2 (Requires that you create a cluster) Another way to obtain cluster JSON skeleton is by navigating to the details of a cluster that has already been created. Under ACTIONS, there is an option to SHOW CLI COMMAND, which allows you to generate a JSON skeleton of your cluster: Obtain CLI Command Syntax from the UI In addition to allowing you to obtain CLI JSON content from the web UI, Cloudbreak allows you to obtain some commands from the UI by using the SHOW CLI COMMAND option. For example: When creating a credential, you can generate `cb credential create` When creating a blueprint, you can generate `cb blueprint create` When adding a recipe, you can generate `cb recipe create` ... View more

Due to a recent change in Google Cloud's handling of requests, all clusters created on Google Cloud Platform with Cloudbreak Cloudbreak 2.4.0 (or earlier 2.x) and 1.16.5 (or earlier 1.x) fail with an error similar to: Infrastructure creation failed. Reason: com.sequenceiq.cloudbreak.cloud.gcp.GcpResourceException: Error during status check: [ resourceType: GCP_DISK, resourceName: hdf-test, stackId: 5, operation: hdftest-s-2-20180323123311 ] [ Cause message: 403 Forbidden { "code" : 403, "errors" : [ { "domain" : "global", "message" : "Required 'compute.globalOperations.get' permission for 'projects/*******/global/operations/operation-1521808391647-56813a099af18-300f33eb-8a28c20c'", "reason" : "forbidden" } ], "message" : "Required 'compute.globalOperations.get' permission for 'projects/*******/global/operations/operation-1521808391647-56813a099af18-300f33eb-8a28c20c'" } ] In order to address the issue, we created a 2.4.1 maintenance release. You can upgrade to Cloudbreak 2.4.1 by using cbd update. For detailed update steps, refer to Cloudbreak 2.4.1 docs. > As a side note, Cloudbreak 2.4.1 also includes a fix for time-based scaling. If you are using Cloudbreak 1.x and do not want to update to Cloudbreak 2.x, you can update to Cloudbreak 1.16.6 by following these steps: 1.Download the CBD binary: curl -Ls public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_1.16.6_$(uname)_x86_64.tgz | sudo tar -xz -C /bin cbd 2.After updating, restart Cloudbreak (must be done from the deployment directory): cbd restart ... View more

This article discusses prerequisites required for Cloudbreak credential and addresses common issues. This article applies to Cloudbreak 2.4.0. When getting started with Cloudbreak, the most common pain point is creating the Cloudbreak credential, or - more precisely - meeting the prerequisites for the Cloudbreak credential. Why do you even need this Cloudbreak credential? Cloudbreak not only runs on your cloud provider account, but once you start using, it provisions resources (such as VMs) that run as part of your account and that you or your organization must pay for. Cloudbreak credential provides the means for Cloudbreak to authenticate with your cloud provider account and provision such resources. The exact details vary depending on the cloud provider: On Azure this is typically done by creating an app registration in Azure Active Directory and providing its details to Cloudbreak. On AWS this typically happens by assigning a proper IAM role to Cloudbreak. Cloudbreak can assume this role in order to create resources on your behalf. On Google Cloud this is typically done by creating a Service Account that Cloudbreak can use to create resources. Azure Credential There are two ways in which Cloudbreak can access your Azure account: interactive and app-based. Interactive The first way is via the interactive credential, where Cloudbreak automated the app and service principal creation and role assignment, so the only input that you need to provide is the Subscription ID and Directory ID. You can definitely use this if you are evaluating Cloudbreak by deploying it on your own Azure account. However, if you are using a corporate account, you will most likely be unable to use the interactive credential at all, because your account must have the "Owner" role (or its equivalent) in order for Cloudbreak to perform the interactive credential creation steps. The error that you will get will be: You don't have enough permissions to assign roles, please contact your administrator Also - just to throw it up there - when using this method, you must be able log in to your Azure account. > This is true for Cloudbreak 2.4.0, but may change in future releases. App-based If you do not meet the requirements for using the interactive credential, you are left with the second option, which is the app-based credential. The advantage of the app-based credential creation is that it allows you to create a credential without logging in to the Azure account, as long as you have been given all the required information. You must provide your Subscription ID and Directory ID (called “Tenant ID” in Cloudbreak UI), you must provide information for your previously created Azure AD application (its ID and key which allows access to it). One tip here is that after registering an Azure application you may have to ask your Azure administrator to perform the step of assigning the "Contributor" role to it: Your account will most likely not have the required permissions and you will get an error. Once your Azure admin performed the role assignment, you can perform the create credential task in Cloudbreak. Related docs: Create Cloudbreak Credential on Azure AWS Credential On AWS, there are two ways for Cloudbreak to authenticate with your AWS account and create resources on your behalf: key-based credential and role-based credential. Key-based To use this, you must be able to provide your AWS access key and secret key pair. If you don’t have these, you may be able to generate a new access and secret key pair from the IAM Console > Users. If you are able to generate these, then this is the easiest way to get started. However, when using a corporate account, you may want to or you may have to use the more complicated role-based credential. Role-based Role-based credential utilizes IAM (Identity Access Management) roles. There are two IAM roles involved: When launching Cloudbreak, you are required to provide an IAM role with AssumeRole policy assigned. The AssumeRole allows Cloudbreak to use the CredentialRole described in the next bullet. After Cloudbreak is running, when you create a Cloudbreak credential, you must provide an IAM Role ARN for another IAM role (which is called CredentialRole in the documentation). This role provides a set of permissions required for provisioning resources. If you chose (or were forced to choose) this approach, you may need to contact your AWS admin to help you out, as you may not be able to create IAM roles. Another tip is that you should make sure to include all the permissions mentioned in the CredentialRole policy definition. If you miss some of them, you will get an access-related error when creating a cluster. Reference: AssumeRole policy definition for Cloudbreak: { "Version": "2012-10-17", "Statement": { "Sid": "Stmt1400068149000", "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": "*" } } Policy definition required for Cloudbreak credential: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudformation:CreateStack", "cloudformation:DeleteStack", "cloudformation:DescribeStackEvents", "cloudformation:DescribeStackResource", "cloudformation:DescribeStacks" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "ec2:AllocateAddress", "ec2:AssociateAddress", "ec2:AssociateRouteTable", "ec2:AuthorizeSecurityGroupIngress", "ec2:DescribeRegions", "ec2:DescribeAvailabilityZones", "ec2:CreateRoute", "ec2:CreateRouteTable", "ec2:CreateSecurityGroup", "ec2:CreateSubnet", "ec2:CreateTags", "ec2:CreateVpc", "ec2:ModifyVpcAttribute", "ec2:DeleteSubnet", "ec2:CreateInternetGateway", "ec2:CreateKeyPair", "ec2:DisassociateAddress", "ec2:DisassociateRouteTable", "ec2:ModifySubnetAttribute", "ec2:ReleaseAddress", "ec2:DescribeAddresses", "ec2:DescribeImages", "ec2:DescribeInstanceStatus", "ec2:DescribeInstances", "ec2:DescribeInternetGateways", "ec2:DescribeKeyPairs", "ec2:DescribeRouteTables", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcs", "ec2:DescribeSpotInstanceRequests", "ec2:DescribeVpcAttribute", "ec2:ImportKeyPair", "ec2:AttachInternetGateway", "ec2:DeleteVpc", "ec2:DeleteSecurityGroup", "ec2:DeleteRouteTable", "ec2:DeleteInternetGateway", "ec2:DeleteRouteTable", "ec2:DeleteRoute", "ec2:DetachInternetGateway", "ec2:RunInstances", "ec2:StartInstances", "ec2:StopInstances", "ec2:TerminateInstances" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "iam:ListRolePolicies", "iam:GetRolePolicy", "iam:ListAttachedRolePolicies", "iam:ListInstanceProfiles", "iam:PutRolePolicy", "iam:PassRole", "iam:GetRole" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "autoscaling:CreateAutoScalingGroup", "autoscaling:CreateLaunchConfiguration", "autoscaling:DeleteAutoScalingGroup", "autoscaling:DeleteLaunchConfiguration", "autoscaling:DescribeAutoScalingGroups", "autoscaling:DescribeLaunchConfigurations", "autoscaling:DescribeScalingActivities", "autoscaling:DetachInstances", "autoscaling:ResumeProcesses", "autoscaling:SuspendProcesses", "autoscaling:UpdateAutoScalingGroup" ], "Resource": [ "*" ] } ] } Related docs: IAM Roles (AWS) Authentication with AWS Create Cloudbreak Credential Google Cloud Credential In order to launch clusters on GCP via Cloudbreak, you must have a Service Account that Cloudbreak can use to create resources. In addition, you must also have a P12 key associated with that account. Usually, a user with an "Owner" role can assign roles to new and existing service accounts from IAM & Admin > IAM. If you are using your own account, you should be able to perform this step, but if you are using a corporate account, you will likely have to contact your Google Cloud admin. The service account must have the following roles enabled: Compute Engine > Compute Image User Compute Engine > Compute Instance Admin (v1) Compute Engine > Compute Network Admin Compute Engine > Compute Security Admin Storage > Storage Admin Note that you must include these exact permissions or else you will run into problems. Related docs: Service Accounts (Google Cloud) Service Account for Cloudbreak Create Cloudbreak Credential General Tips 1. Although the UI option and CLI command for creating a Cloudbreak credential appear to be simple, the related prerequisites are actually complex and, if you are using a corporate cloud account, you may have to contact your cloud account administrator to perform some of the prerequisite steps for you. 2. When meeting the prerequisites for Cloudbreak credential (for example trying to create an IAM role on AWS, trying to create an app registration on Azure, trying to create a Service Account on Google Cloud), an error related to permissions/authorization/roles or an inability to access certain options may mean that you do not have the permissions to perform certain actions and you may have to ask your cloud admin for help. 3. Once the credential has been created and you are trying to create a cluster, an error related to permissions/authorization/roles usually means that the scope of permissions assigned to Cloudbreak (via IAM role on AWS, Contributor role equivalent on Azure, Service Account on Google Cloud) is insufficient. When the scope of permissions is insufficient, you may experience other issues, such as being unable to see provider regions and instance types in the create cluster UI wizard or CLI. As far as I know, Cloudbreak 2.4.0 does not check the permissions provided in the policy assigned to the IAM role (on AWS) or Service Account (Google Cloud). On Azure, on the other hand, these checks are in place. ... View more

Cloudbreak 2.4.0 is now live! What is Cloudbreak: Cloudbreak simplifies cluster provisioning on public cloud infrastructure platforms Microsoft Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP), and on the private cloud infrastructure platform OpenStack. Here are the highlights of what’s new in Cloudbreak 2.4.0 compared with Cloudbreak 1.16.x: New UI/UX Cloudbreak 2.4.0 introduces a new user interface. All major options are now easily accessible from the collapsible navigation menu. All UI options and wizards have been redesigned in order to make cluster creation and management more intuitive. The following screenshot shows the cluster dashboard: The following screenshot shows cluster details page: > Note that autoscaling is available from the Autoscaling tab. > Note the Show CLI Command available from Actions menu. This allows you to generate a CLI template of the cluster. A similar option is available on the last page of the create cluster wizard. The following screenshot shows the create cluster wizard: > Note the Basic/Advanced toggle button, which allows you to switch between the basic and advanced wizard view. In the Settings you can select which view you would like to see by default. New CLI Cloudbreak 2.4.0 introduces the new CLI tool, which replaces Cloudbreak Shell. All commands start with a singular object followed by an action, for example, blueprint create and blueprint list. To download the CLI, select Download CLI from the navigation pane. The following commands are available: blueprint blueprint related operations cloud information about cloud provider resources cluster cluster related operations configure configure the server address and credentials used to communicate with this server credential credential related operations imagecatalog imagecatalog related operations recipe recipe related operations help, h Shows a list of commands or help for one command Refer to Install CLI, Get Started with the CLI, and CLI Reference. Support for Kerberos Creating Kerberos-enabled clusters is supported: Use an existing MIT KDC or Active Directory for production deployments. Use a test KDC for evaluation purposes. Refer to Enabling Kerberos Security. Configuring an External Database for Cloudbreak Using an external RDBMS for Cloudbreak is supported and recommended for production environments. For configuration instructions, refer to Configuring External Cloudbreak Database. Migrating Cloudbreak Instance Migrating Cloudbreak from one machine to another is supported. For migration instructions, refer to Moving a Cloudbreak Instance. Prewarmed Images To accelerate cluster creation, CLoudbreak 2.4.0 introduces prewarmed images, which include the operating system, as well as the default version of Ambari and HDP. By default, Cloudbreak 2.4 launches clusters from these prewarmed images, instead of using base images (which were used by default in earlier versions of Cloudbreak). Default base images are still available in case you would like to use different Ambari and HDP versions than those provided with prewarmed images. For more information, refer to Prewarmed and Base Images. Providing Your Own SDK Providing your own JDK on a custom base image is supported. For instructions, refer to "Advanced topics" in the https://github.com/hortonworks/cloudbreak-images repository. Generating CLI Templates After specifying the parameters for your cluster in the Cloudbreak web UI, you can copy the content of the CLI JSON file that can be used to create a cluster via Cloudbreak CLI. For more information, refer to Obtain Cluster JSON Template from the UI. Furthermore, Cloudbreak web UI includes an option in the UI which allows you to generate the createcommand for resources such as credentials, blueprints, clusters, and recipes. For more information, refer to Obtain CLI Command from the UI. New Recipe Types New types of recipes are introduced: PRE-AMBARI-START (new, useful for configuring Ambari prior to start) POST-AMBARI-START (formerly known as PRE) POST-CLUSTER-INSTALL (formerly known as POST) PRE-TERMINATION (new, useful for cluster cleanup pre-termination tasks) Refer to Recipes documentation. Disabling Cloud Providers You can hide cloud providers available in Cloudbreak by adding the CB_ENABLEDPLATFORMS environment variable in Profile and setting it to the provider(s) that you would like to have available. For more information, refer to Disable Providers. Support for Ambari 2.6.1 (Specifically 2.6.1.3+) Cloudbreak 2.4.0 introduces support for Ambari 2.6 and uses Ambari 2.6.1.3 by default. If you would like to use Ambari 2.6, you must use Ambari version 2.6.1.3 or newer. If you decide to use Ambari 2.6.1, you should be aware of the following: Ambari 2.6.1 or newer does not install the mysqlconnector; therefore, when creating a blueprint for Ambari 2.6.1 or newer do not include the MYSQL_SERVER component for Hive Metastore in your blueprint. Instead, you have two options described in Creating a Blueprint. If you would like to use Oozie, you must manually install Ext JS. The steps are described in Cannot Access Oozie Web UI. To enable LZO compression in your HDP cluster, you must check the "Enable Ambari Server to download and install GPL Licensed LZO packages?" during cluster creation. The option is available under Security > Prewarmed and Base Images. Ambari Master Key Cloudbreak 2.4.0 allows you to specify the Ambari Master Key. The Ambari Server Master Key is used to configure Ambari to encrypt database and Kerberos credentials that are retained by Ambari as part of the Ambari setup. The option is available on the Security page of the create cluster wizard. Get Started with Cloudbreak 2.4.0 GA To get started on your chosen platform, refer to Cloudbreak 2.4.0 documentation. > Note that Cloudbreak 2.4.0 documentation was written from scratch, rather than being based on Cloudbreak 1.16.x documentation. The steps for meeting the prerequisites, launching Cloudbreak, and creating a Cloudbreak credentials are described in detail, including numerous screenshots. They are appropriate for users who want to get started with Cloudbreak but are not familiar with the AWS, Azure, and Google Cloud cloud providers. Get started now with Cloudbreak 2.4.0! ... View more

Cloudbreak is now available directly from the Azure Marketplace. The goal of this article is to help you get started: quickly launch Cloudbreak from the Azure Marketplace, log in to the Cloudbreak UI and create a Cloudbreak credential using the interactive method. Let's get started! Update This article was written in 2017. As of today, Azure Marketplace no longer includes the latest version of Cloudbreak. Fo the latest version use this quickstart. Overview We will be performing the following steps: Meet the Prerequisites. Launch Cloudbreak. Log in to the Cloudbreak UI. Create a Cloudbreak credential using the interactive method. Meet the Prerequisites 1.In order to launch Cloubdreak on the Azure Marketplace, you must have a Microsoft Azure account. If you don't have an account, you can set it up at https://azure.microsoft.com. 2.In order to provision clusters on Azure, Cloudbreak must be able to assume a sufficient Azure role ("Owner" or "Contributor") via Cloudbreak credential: Your account must have the "Owner" role in the subscription in order to create a Cloudbreak credential using the interactive credential method. Your account must have the "Contributor" role (or higher) in the subscription in order to create a Cloudbreak credential using the app-based credential method. To check the roles that your subscription has, in your Azure account navigate to Subscriptions. Launch Cloudbreak 1.Log in to your Azure Portal. 2.From the services menu, select Marketplace. 3.Search for “Cloudbreak” and click on “Cloudbreak for Hortonworks Data Platform”. 4.Click on the Create button under the listing to access the wizard. 5.Provide the following parameters on the Basics page of the wizard: Administrator email address: Enter your email address. You will use this email address to log in to the Cloudbreak web UI. Administrator user password and Confirm password: Enter a password. You will use this password to log in to the Cloudbreak web UI. VM Username: Enter your chosen username. If needed, you will use it to SSH to the VM on which Cloudbreak is running. VM SSH public key: Paste your public SSH key. You can copy your public SSH key to the clipboard using pbcopy < ~/.ssh/id_rsa.pub. If needed, you can generate a new SSH keypair using ssh-keygen -t rsa -b 4096 -C "your_email@example.com". Subscription: Select your existing subscription. Resource group: Azure uses resource groups to logically organize related resources. Create a new resource group for your Cloudbreak deployment. Location: Select the Azure region in which you would like to launch Cloudbreak. You can launch Cloudbreak and provision your clusters in all regions supported by Microsoft Azure. Clusters created via Cloudbreak can be in the same or different region as Cloudbreak; when you launch a cluster, you select the region in which to launch it. 6. Click OK to move to the next page in the wizard. 7.Provide the following parameters on the Advanced settings page of the wizard: Controller Instance Type: This is the VM type that will be used to launch Cloudbreak. The default 1x Standard DS2 v2 is sufficient. Allow connections to the cloud controller from this address range or default tag (e.g. Internet): Use the following tool to determine the CIDR based on your IP address: or you can enter “Internet” to allow connections from all. Virtual network: By default, a new virtual network called “cbdeployerVnet” is created. You can also select an existing network. Subnets: By default a subnet called “cbdeployerSubnet” is created. If you selected an existing network, you must configure your own subnet. 8. Click OK to move to the next page in the wizard. 9.On the Summary page of the wizard, confirm that all the information looks correct and click OK to proceed. 10.On the Buy page of the wizard, click on Purchase. This starts the launch process. 11.You will be redirected to your dashboard. On your dashboard and in the notifications, you will see that Cloudbreak is being deployed: Log in to the Cloudbreak UI 1.Once the deployment is complete, you will see “Deployments succeeded” in your notifications and a new tile with resources will be added to your dashboard: 2.Click on the tile to navigate to the resource group. 3.Click on Deployments. 4.Click on hortonworks.cloudbreak-for-hortonworks-data-platf-<>. 5. Copy the URL from Outputs > LOGINURL. 6. Paste the URL into your browser. 7.Your browser will warn you that the connection is unsafe. Proceed to the UI. 8. Log in with the administrator email address and password created in the first step of the Azure wizard. Create a Cloudbreak Credential 1.In the Cloudbreak web UI, click on +create credential. 2. The wizard will appear. Click Next. 3. On the second page of the credential wizard, provide a name for your credential. By default, an interactive credential is pre-selected, your public SSH key (which will be used to SSH to cluster VMs) is pre-entered, and Cloudbreak uses your existing “Contributor” role. If you would like to change any of these defaults, refer to the Cloudbreak on Azure Marketplace documentation. 4. Click on Next. 5. On the last page of the wizard, copy the code and click on Azure login. 6. You will be redirected to the Device Login page. Paste the code that you copied earlier. 7.Click Continue and then select your account. 8.Once you see the following page, you may close the window: 9. Go back to the Cloudbreak UI and make sure that the credential that you created is available and pre-selected at the top of the page: Congratulations! Now you can start creating clusters! For more information refer to the Cloudbreak on Azure Marketplace documentation. ... View more