If MySQL Driver does not exist by default, or not installed when MySQL is installed, then the provisioned cluster has the following errors. In this case, create a Recipe in Cloudbreak to install MySQL jdbc driver at pre-ambari-start time: During Create Cluster, use Advance option. Pick the recipe pk-install-mysql-driver-pre from the drop down and click Attach. This will ensure that the jdbc driver is installed before Ambari is started. This should then result in a clean install. ... View more

If you see errors related to YARN Registry DNS Bind Port. Where the YARN Registry DNS is stopped, most probably due to a port conflict. Then go into the Advanced Configurations. Look for the parameter RegistryDNS Bind Port and change is to a port that does not have a conflict. I changed it from 53 to 553. Save configuration changes. Restart all components that are impacted. This should take care of the issue and YARN Registry DNS will start successfully. ... View more

Launch Cloudbreak on AWS Before launching Cloudbreak on AWS, review and meet the prerequisites. Meet the Prerequisites Before launching Cloudbreak on AWS, you must meet the following prerequisites. AWS Account AWS Region SSH Key Pair Key Based Authentication AWS Account In order to launch Cloudbreak on AWS, you must log in to your AWS account. If you don't have an account, you can create one at https://aws.amazon.com/. AWS Region Decide in which AWS region you would like to launch Cloudbreak. The following AWS regions are supported: SSH Key Pair Import an existing Key Pair or generate a new Key Pair in the AWS region which you are planning to use for launching Cloudbreak and clusters. You need this SSH Key Pair to SSH to the Cloudbreak instance and Start Cloudbreak. To do this use the following steps. Navigate to the Amazon EC2 console at https://console.aws.amazon.com/ec2/. Check the region listed in the top right corner to make sure that you are in the correct region. In the left pane, find NETWORK AND SECURITY and click Key Pairs. Do one of the following: To generate a new Key Pair: Click Create Key Pair to create a new key pair Your private Key file will be automatically downloaded onto your computer. Make sure to save it in a secure location. You will need it to SSH to the cluster nodes. You may want to change access settings for the file using chmod 400 my-key-pair.pem. To import an existing Public Key: Click Import Key Pair to upload an existing Public Key Select the Public Key and click Import. Make sure that you have access to its corresponding Private Key. Key Based Authentication If you are using key-based authentication for Cloudbreak on AWS, you must be able to provide your AWS Access Key and Secret Key pair. Cloudbreak will use these keys to launch the resources. You must provide the Access and Secret Keys later in the Cloudbreak web UI later when creating a credential. If you choose this option, all you need to do at this point is check your AWS account and make sure that you can access this Key Pair. You can generate new access and secret keys from the IAM Console. To do this go to the IAM Service: In the left pane click on Users > Select a user Click on the Security credentials tab Creat Access Key or use an existing Access Key. There is a limit of two. To create a new user go to IAM Service: 1.In the left pane click on Users > Select Add User 2.Enter a user name, select Programming access and then click on Next: Permissions 3.For Set Permissions keep the defaul "Add usert to group" 4.In Add User to group select all the groups then click Next:Review 5.Review your choices and click on Create User 6.If your user has been created successfully you will see a similar image as below 7.Once you are done with these steps, you are now ready to launch Cloudbreak. Cloudbreak from quickstart template Follow documentation provided here to install cloudbreak. Next go to:Cloudbreak on AWS. Go straight to "Log into the Cloudbreak application" ... View more

Use S3 as storage for Zeppelin Notebooks. Step 1. Use external storage to point to S3 bucket in Cloudbreak advance options. This will use S3access profile and AWS credentials. CB takes care of that set up. In addition to that Step 2 Change these 3 inzeppelin-env.sh export ZEPPELIN_NOTEBOOK_S3_BUCKET=yourBucketName export ZEPPELIN_NOTEBOOK_S3_ENDPOINT="http://s3.amazonaws.com/yourBucketName" export ZEPPELIN_NOTEBOOK_S3_USER=admin Step 3 Change this 1 inzeppelin-site.xml Point zeppelin.notebook.storage to org.apache.zeppelin.notebook.repo.S3NotebookRepo Detailed Steps below 1.Complete AWS pre-requisites 2.Create AWS credentials in Cloudbreak 3.Launch HDP cluster on AWS using Cloudbreak. 4.Enable Zeppelin Storage on S3. 3. Launch HDP Cluster on AWS using Cloudbreak. Not all the screenshots are included, only capturing screenshots that focus on some key Advance features that enable the required Zeppelin Storage. a.Use the Advance tab on Cloudbreak. b.Cloudbreak uses AWS credentials that will provide the necessary AWS Key and Secrect Access Key for S3 Storage setup. a.Provide an instance profile created in AWS that has access to your S3 b.Provide your bucket name for base storage 4.Enable Zeppelin Storage on S3. Once Ambari starts and all services are started we need to make some configuration changes to enable Zeppelin. In the zeppelin-config change the following properties: zeppelin_notebook.s3.bucket zeppelin_notebook.s3.user zeppelin_notebook.storage Or you could change them in zeppelin-env: export ZEPPELIN_NOTEBOOK_S3_BUCKET=bucketName export ZEPPELIN_NOTEBOOK_S3_ENDPOINT="http://s3.amazonaws.com/bucketName" export ZEPPELIN_NOTEBOOK_S3_USER=admin Here is an example path: bucket/user/notebook/2A94M5J1Z/note.json Now when you create and save Notebooks in Zeppelin, it will save in S3. You will be able to see the notebooks in your AWS portal, in your S3 bucket. Zeppelin notebooks use 9 character hash as the name of the folder and note.json file in that folder. ... View more

Launching Cloudbreak cluster on Azure Before launching Cloudbreak cluster on Azure, review and meet the prerequisites. Meet the Prerequisites Before launching Cloudbreak on Azure, you must meet the following prerequisites. Azure Account Azure Region SSH Key Pair Authentication Subscription Id Tenant ID Application ID Azure Account In order to launch Cloudbreak on the Azure, log in to your existing Microsoft Azure account. If you don't have an account, you can set it up at https://azure.microsoft.com. Azure Region Decide in which Azure region you would like to launch Cloudbreak. You can launch Cloudbreak and provision your clusters in all regions supported by Microsoft Azure. Create an SSH key pair When launching Cloudbreak, you will be required to provide your public SSH key. If needed, you can generate a new SSH key pair: On MacOS X and Linux using ssh-keygen -t rsa -b 4096 -C "your_email@example.com" On Windows using PuTTygen On Azure shell. Launch azure shell via a browser https://shell.azure.com/ and Create an SSH key pair Use the ssh-keygen command to generate SSH public and private key files that are by default created in the ~/.ssh directory. Authentication For Subscription ID Search for Subscription Service Overview Subscription ID For Tenant ID Look for Azure Active Directory Properties Directory ID is Tenant ID For Application ID Look for Azure Active Directory App registration Identify the application for cloudbreak Pick the Application Id associated If an Application is not already there, then you can create one. The key thing is that this Application will need Contributor Role associated and normally only your Enterprise Account owner will be able to grant that access. Here is a screenshot for creating the Application. For App Key Look for Azure Active Directory and click App registration Identify the application for cloudbreak Click on Settings Keys Generate new Key or access an existing Key. This key is visible the first time its created only, so please makes ure to save it in a safe place. ... View more

Cloudbreak on AWS Before launching Cloudbreak on AWS, review and meet the prerequisites. Meet the Prerequisites Before launching Cloudbreak on AWS, you must meet the following prerequisites. AWS Account AWS Region SSH Key Pair Key Based Authentication AWS Account In order to launch Cloudbreak on AWS, you must log in to your AWS account. If you don't have an account, you can create one at https://aws.amazon.com/. AWS Region Decide in which AWS region you would like to launch Cloudbreak. The following AWS regions are supported: SSH Key Pair Import an existing Key Pair or generate a new Key Pair in the AWS region which you are planning to use for launching Cloudbreak and clusters. You need this SSH Key Pair to SSH to the Cloudbreak instance and Start Cloudbreak. To do this use the following steps. Navigate to the Amazon EC2 console at https://console.aws.amazon.com/ec2/. Check the region listed in the top right corner to make sure that you are in the correct region. In the left pane, find NETWORK AND SECURITY and click Key Pairs. Do one of the following: To generate a new Key Pair: Click Create Key Pair to create a new key pair Your private Key file will be automatically downloaded onto your computer. Make sure to save it in a secure location. You will need it to SSH to the cluster nodes. You may want to change access settings for the file using chmod 400 my-key-pair.pem. To import an existing Public Key: Click Import Key Pair to upload an existing Public Key Select the Public Key and click Import. Make sure that you have access to its corresponding Private Key. Key Based Authentication If you are using key-based authentication for Cloudbreak on AWS, you must be able to provide your AWS Access Key and Secret Key pair. Cloudbreak will use these keys to launch the resources. You must provide the Access and Secret Keys later in the Cloudbreak web UI later when creating a credential. If you choose this option, all you need to do at this point is check your AWS account and make sure that you can access this Key Pair. You can generate new access and secret keys from the IAM Console. To do this go to the IAM Service: In the left pane click on Users > Select a user Click on the Security credentials tab Creat Access Key or use an existing Access Key. There is a limit of two. To create a new user go to IAM Service: 1.In the left pane click on Users > Select Add User 2.Enter a user name, select Programming access and then click on Next: Permissions 3.For Set Permissions keep the defaul "Add usert to group" 4.In Add User to group select all the groups then click Next:Review 5.Review your choices and click on Create User 6.If your user has been created successfully you will see a similar image as below 7.Once you are done with these steps, you are now ready to create cloud credentials in your Cloudbreak application. Log into the Cloudbreak application 1. Confirm the security exception Access Cloudbreak via a browser: https://cloudbreak-crashcourse.com/sl/ You will likely see a browser warning when you first open the Ambari UI. That is because we are using self-signed certificates. Click on Advance and then proceed to cloudbreak server. 2. login page Cloudbreak login page. Credentials will be provided for these services by the instructor. Once you log in as a first time user, your screen will look like this. Click on Go To Credentials . 3. Create Cloudbreak Credentials The first thing you need are credentials to a cloud provider of your choice. Click on Credentials in the right menu Then click on Create Credentials Pick the cloud provider of your choice Pick a type of credential, Role based or Key based. For the purpose of this tutorial pick Role based. Provide a name and description for the credential. Get the Access Key and Secret Key from previous section. If everything has gone well, then the credentials are created. ... View more

Cloudbreak launches clusters on the cloud in 3 easy steps: Pick a Blueprint: Cloudbreak uses Ambari Blueprints to have declarative Hadoop cluster definition. Blueprints can be designed for specialized applications and workloads (such as Data Science or IoT Apps). Cloudbreak includes a few default Blueprints for common cluster configurations but you can always upload your own Blueprint to build the cluster just the way you like it. Choose a Cloud: Cloudbreak is configured to work with cloud infrastructure resources (such as servers, network setup and security options). Choose the cloud infrastructure you want to use for the cluster. Launch Cluster: In this step, Cloudbreak obtains the chosen cloud infrastructure platform, installs Apache Ambari and applies the desired Blueprint. The result: your cluster is launched and ready to go! Prerequisites a. Before launching Cloudbreak on AWS, you must meet the AWS prerequisites. b. Before launching Cloudbreak on Azure, you must meet the Azure prerequisites. b. Before launching Cloudbreak on GCP, you must meet the Google Cloud prerequisites. ... View more