Objectives
The release of Cloudbreak 2.7 enables you to deploy Hortonworks Data Flow (HDF) clusters. Currently there are two HDF cluster types supported: Flow Management (NiFi) and Messaging Management (Kafka). Cloudbreak expects HDF clusters to be deployed with security (LDAP, SSL). However, for testing purposes, many people would like to deploy a cluster without having to go through the steps of setting up SSL, LDAP, etc. Therefore, we'll need to modify the default HDF Flow Management blueprint to loosen the security configuration. This is not recommended for production use cases.
This tutorial is designed to walk you through the process of of deploying an HDF 3.1 Flow Management Cluster on AWS using Cloudbreak 2.7 using a custom blueprint.
Prerequisites
- You should already have an installed version of Cloudbreak 2.7.
- You can find the documentation on Cloudbreak here: Cloudbreak Documentation
- You can find an article that walks you through installing a local version of Cloudbreak with Vagrant and Virtualbox here: HCC Article
- You should have an AWS account with appropriate permissions.
- You can read more about AWS permissions here: Cloudbreak Documentation
- You should already have created your AWS credential in Cloudbreak.
Scope
This tutorial was tested in the following environment:
- Cloudbreak 2.7.0
- AWS (also works on Azure and Google)
Steps
1. Create New HDF Blueprint
Login to your Cloudbreak instance. In the left menu, click on Blueprints. Cloudbreak will display a list of built-in and custom blueprints. Click on the Flow Management: Apache NiFi, Apache NiFi Registry blueprint. you should see something similar to the following:
Now click on the RAW VIEW tab. You should see something similar to the following:
Now we need to copy the raw JSON from this blueprint. We need to make some modifications. Copy and paste the blueprint into your favorite text editor.
Change the blueprint_name line to "blueprint_name": "hdf-nifi-no-kerberos",. This is the name of the blueprint and it must be unique from other blueprints registered in Cloudbreak.
In the nifi-properties section we need to add a new line. We are going to add "nifi.security.user.login.identity.provider": "". This change tells NiFi not to use an Identity Provider. Change this:
{
"nifi-properties": {
"nifi.sensitive.props.key": "changemeplease",
"nifi.security.identity.mapping.pattern.kerb": "^(.*?)@(.*?)$",
"nifi.security.identity.mapping.value.kerb": "$1",
}
},
to this:
{
"nifi-properties": {
"nifi.sensitive.props.key": "changemeplease",
"nifi.security.identity.mapping.pattern.kerb": "^(.*?)@(.*?)$",
"nifi.security.identity.mapping.value.kerb": "$1",
"nifi.security.user.login.identity.provider": ""
}
},
In the nifi-ambari-ssl-config section we need to change the nifi.node.ssl.isenabled settings from true to false. This change disables SSL between the NiFi nodes. Change this:
"nifi-ambari-ssl-config": {
"nifi.toolkit.tls.token": "changemeplease",
"nifi.node.ssl.isenabled": "true",
"nifi.toolkit.dn.prefix": "CN=",
"nifi.toolkit.dn.suffix": ", OU=NIFI"
}
to this:
"nifi-ambari-ssl-config": {
"nifi.toolkit.tls.token": "changemeplease",
"nifi.node.ssl.isenabled": "false",
"nifi.toolkit.dn.prefix": "CN=",
"nifi.toolkit.dn.suffix": ", OU=NIFI"
}
In the nifi-registry-ambari-ssl-config section we need to change the nifi.registry.ssl.isenabled settings from true to false. This change disables SSL for the NiFi Registry. Change this:
"nifi-registry-ambari-ssl-config": {
"nifi.registry.ssl.isenabled": "true",
"nifi.registry.toolkit.dn.prefix": "CN=",
"nifi.registry.toolkit.dn.suffix": ", OU=NIFI"
}
to this:
"nifi-registry-ambari-ssl-config": {
"nifi.registry.ssl.isenabled": "false",
"nifi.registry.toolkit.dn.prefix": "CN=",
"nifi.registry.toolkit.dn.suffix": ", OU=NIFI"
}
Under host_groups and Services we need to remove the NIFI_CA entry. This change removes the NiFi Certificate Authority. Change this:
"host_groups": [
{
"name": "Services",
"components": [
{
"name": "NIFI_CA"
}, {
"name": "NIFI_REGISTRY_MASTER"
},
to this:
"host_groups": [
{
"name": "Services",
"components": [
{
"name": "NIFI_REGISTRY_MASTER"
},
The complete blueprint looks like this:
{
"Blueprints": {
"blueprint_name": "hdf-nifi-no-kerberos",
"stack_name": "HDF",
"stack_version": "3.1"
},
"configurations": [
{
"nifi-ambari-config": {
"nifi.security.encrypt.configuration.password": "changemeplease",
"nifi.max_mem": "1g"
}
},
{
"nifi-properties": {
"nifi.sensitive.props.key": "changemeplease",
"nifi.security.identity.mapping.pattern.kerb": "^(.*?)@(.*?)$",
"nifi.security.identity.mapping.value.kerb": "$1",
"nifi.security.user.login.identity.provider": ""
}
},
{
"nifi-ambari-ssl-config": {
"nifi.toolkit.tls.token": "changemeplease",
"nifi.node.ssl.isenabled": "false",
"nifi.toolkit.dn.prefix": "CN=",
"nifi.toolkit.dn.suffix": ", OU=NIFI"
}
},
{
"nifi-registry-ambari-config": {
"nifi.registry.security.encrypt.configuration.password": "changemeplease"
}
},
{
"nifi-registry-properties": {
"nifi.registry.sensitive.props.key": "changemeplease",
"nifi.registry.security.identity.mapping.pattern.kerb": "^(.*?)@(.*?)$",
"nifi.registry.security.identity.mapping.value.kerb": "$1"
}
},
{
"nifi-registry-ambari-ssl-config": {
"nifi.registry.ssl.isenabled": "false",
"nifi.registry.toolkit.dn.prefix": "CN=",
"nifi.registry.toolkit.dn.suffix": ", OU=NIFI"
}
}
],
"host_groups": [
{
"name": "Services",
"components": [
{
"name": "NIFI_REGISTRY_MASTER"
},
{
"name": "METRICS_COLLECTOR"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "METRICS_GRAFANA"
},
{
"name": "ZOOKEEPER_CLIENT"
}
],
"cardinality": "1"
},
{
"name": "NiFi",
"components": [
{
"name": "NIFI_MASTER"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "ZOOKEEPER_CLIENT"
}
],
"cardinality": "1+"
},
{
"name": "ZooKeeper",
"components": [
{
"name": "ZOOKEEPER_SERVER"
},
{
"name": "METRICS_MONITOR"
},
{
"name": "ZOOKEEPER_CLIENT"
}
],
"cardinality": "3+"
}
]
}
Save the updated blueprint to a file. Click on the CREATE BLUEPRINT button. You should see the Create Blueprint screen.
Enter the name of the new blueprint, something helpful such as hdf-nifi-no-kerberos. Click on the Upload JSON File button and upload the blueprint you just saved. You should see the new blueprint you created.
2. Create Cluster
In the left menu, click on Clusters. Cloudbreak will display configured clusters. Click the CREATE CLUSTER button. Cloudbreak will display the Create Cluster wizard
a. General Configuration
By default, the General Configuration screen is displayed using the BASIC view. The ADVANCED view gives you more control of AWS and cluster settings, to include features such as tags. You can change your view to ADVANCED manually or you can change your Cloudbreak preferences to show ADVANCED view by default. We will use the BASIC view.
- Credential: Select the AWS credential you created. Most users will only have 1 credential per platform which will be selected automatically.
- Cluster Name: Enter a name for your cluster. The name must be between 5 and 40 characters, must start with a letter, and must only include lowercase letters, numbers, and hyphens.
- Region: Select the region in which you would like to launch your cluster.
- Platform Version: Cloudbreak currently defaults to HDP 2.6. Select the dropdown arrow and select
HDF 3.1. - Cluster Type: As mentioned previously, there are two supported cluster types. Make sure select the blueprint you just created.
Click the green NEXT button.
c. Hardware and Storage
Cloudbreak will display the Hardware and Storagescreen. On this screen, you have the ability to change the instance types, attached storage and where the Ambari server will be installed. As you you can see, we will deploy 1 NiFi and 1 Zookeeper node. In a production environment you would typically have at least 3 Zookeeper nodes. We will use the defaults.
Click the green NEXT button.
d. Gateway Configuration
Cloudbreak will display the Gateway Configuration screen. On this screen, you have the ability to enable a protected gateway. This gateway uses Knox to provide a secure access point for the cluster. Cloudbreak 2.7 does not currently support configuring Knox for HDF. We will leave this option disabled.
Click the green NEXT button.
e. Network
Cloudbreak will display the Network screen. On this screen, you have the ability to specify the Network, Subnet, and Security Groups. Cloudbreak defaults to creating new configurations. For production use cases, we highly recommend creating and refining your own definitions within the cloud platform. You can tell Cloudbreak to use those via the drop down menus. We will use the default options to create new configurations.
Because we are using a custom blueprint which disables SSL, we need to update the security groups with correct ports for the NiFi and NiFi Registry UIs. In the SERVICES security group, add the port 61080 with TCP. Click the + button to add the rule. In the NIFI security group, add the port 9090 with TCP. Click the + button to add the rule.
You should see something similar the following:
Click the green NEXT button.
f. Security
Cloudbreak will display the Security screen. On this screen, you have the ability to specify the Ambari admin username and password. You can create a new SSH key or selecting an existing one. And finally, you have the ability to enable Kerberos on the cluster. We will use admin for the username and BadPass#1 for the password. Select an existing SSH key from the drop down list. This should be a key you have already created and have access to the corresponding private key. We will NOT be enabling Kerberos, so uncheck the Enable Kerberos Security checkbox.
You have the ability to display a JSON version of the blueprint. You also have the ability display a JSON version of the cluster definition. Both of these can be used with Cloudbreak CLI to programatically automate these operations.
Click the green CREATE CLUSTER button.
g. Cluster Summary
Cloudbreak will display the Cluster Summary page. It will generally take between 10-15 minutes for the cluster to be fully deployed. As you can see, this screen looks similar to and HDP cluster. The big difference is the Blueprint and HDF Version.
Click on the Ambari URL to open the Ambari UI.
h. Ambari
You will likely see a browser warning when you first open the Ambari UI. That is because we are using self-signed certificates.
Click on the ADVANCED button. Then click the link to Proceed.
You will be presented with the Ambari login page. You will login using the username and password you specified when you created the cluster. That should have been admin and BadPass#1. Click the green Sign In button.
You should see the cluster summary screen. As you can see, we have a cluster with Zookeeper, NiFi, and the NiFi Registry.
Click on the NiFi service in the left hand menu. Now you can access the Quick Links menu for a shortcut to the NiFi UI.
You should see the NiFi UI.
Back in the Ambari UI, click on the NiFi Registry service in the left hand menu. Now you can access the Quick Links menu for a shortcut to the NiFi Registry UI.
You should see the NiFi Registry UI.
Review
If you have successfully followed along with this tutorial, you should have created a Flow Management (NiFi) cluster on AWS using a custom blueprint. This cluster has SSL and LDAP configurations disabled for rapid prototyping abilities.
