- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Deploying multinode HDP with puppet
- Labels:
-
Apache Ambari
-
Apache Hadoop
-
Apache HBase
Created ‎01-27-2017 12:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Dear colleagues,
I am trying to deploy multinode HDP with puppet, but have some troubles with services preconfiguration.
Regarding ambari install and machines configuration everything is understandable:
1) Setup ssh, libraries, shares etc. using puppet.
2) Install ambari, also puppet - https://forge.puppetlabs.com/vzach/ambari
However the problem appears when configuring services. For example, hbase, MR, ozzie, etc. For this I had not found any puppet libraries. I`ve seen that there is possible to manage a cluster via blueprints or ambari shell. Also had found that there is some limited support of bigtop. The bigtop seems to be more capable, but had not found any description of using it with ambari.
Would be very grateful for some help this question.
Created ‎01-27-2017 08:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In addition to answer given by @Josh Elser. I would like to share my tutorials on Ambari Blueprints to help you get started quickly :
How to install single node HDP cluster using Ambari blueprint:
How to install multi node HDP cluster using Ambari blueprint:
How to install multi node HDP cluster with Namenode HA using Ambari blueprint:
How to install single node HDP cluster with Kerberos authentication using Ambari blueprint:
How to install multinode HDP cluster with Kerberos authentication using Ambari blueprint:
Created ‎01-27-2017 04:55 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I would strongly recommend the use of Ambari to manage your HDP clusters.
If you need to automate the installation of multiple HDP clusters, Ambari blueprints are a very nice feature to reduce the cost of re-installation.
Created ‎01-27-2017 07:28 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I`ve found only curl or some custom app for deploying configuration via blueprints. Is there any other way of deploying it?
Created ‎01-27-2017 08:07 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As Josh already mentioned, the best way to deploy HDP clusters is Ambari. Bigtop is a packaging project that has some ability to deploy unmanaged cluster for testing purposes. Ambari on the other hand is the cluster managing software. It may come that one day we will see Ambari as one of the packages in Bigtop, but even in this case the deployment of cluster will be done by Ambari blueprints.
Created ‎01-27-2017 08:14 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In addition to answer given by @Josh Elser. I would like to share my tutorials on Ambari Blueprints to help you get started quickly :
How to install single node HDP cluster using Ambari blueprint:
How to install multi node HDP cluster using Ambari blueprint:
How to install multi node HDP cluster with Namenode HA using Ambari blueprint:
How to install single node HDP cluster with Kerberos authentication using Ambari blueprint:
How to install multinode HDP cluster with Kerberos authentication using Ambari blueprint:
Created ‎01-27-2017 10:22 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Many thanks. I had already seem them. Is there any cli rather then curl utility for deploying blueprints?
Created ‎01-30-2017 05:31 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I would follow Josh's recommendation to manage the cluster once ambari is installed. Utilizing a configuration management tool like puppet is still useful to do all the preconfigurations before ambari is utilized.
Created ‎01-31-2017 10:12 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Faced several issues when making the deployment:
1. Ambari puppet configurations that can be found on puppetforge are tested only for CentOS and still some features are not supported. I am using debian. Trying to test and make some rewrites for debian and fix some issues.
2. It would be nice to have some utility for managing blueprints (something similar to StackMaster for managing Amazon CloudFormation stacks). Had been thinking about such utility, but it will happen after first point.
Created ‎02-01-2017 10:17 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Faced some question of consistency. Is this configuration https://forge.puppet.com/unibet/ambari fully consistent with latest ambari version?
Created ‎02-07-2017 10:32 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Completed with some basic puppet setup. It required rewriting of some code from https://forge.puppet.com/unibet/ambari. Also found out that following puppet configuration does not support ssl. Trying to implement it, but had not found any manual how to configure ssl for ambari not using "ambari-server setup-security" as specified here https://community.hortonworks.com/articles/39865/enabling-https-for-ambariserver-and-troubleshootin..... I assume that only options needed to make ssl working are:
1) Keystore for ambari services:
"
ssl.trustStore.path=<path-to-keystore-file>
ssl.trustStore.type=jks
ssl.trustStore.password=<keystore-password-here>
"
2) HTTPS options:
"
client.api.ssl.port=8443
api.ssl=true
client.api.ssl.cert_name=https.crt
client.api.ssl.key_name=https.key
"
Isn`t it simpler to use nginx with ssl in front of ambari to provide https?
Just tried to make a manual setup with generating certificates and filling the needed fields in /etc/ambari-server/conf/ambari.properties, afterwards started "ambari-server setup -s", but got "ambari-server restart" failed with error about problem with certificates and truststore. Seems it makes some transformations in process of running setup.
Found out that there are some options for ambari-server setup-security option:
1) Configure https: "ambari-server setup-security --security-option=setup-https --api-ssl=true --import-cert-path=/etc/ambari-server/certs/localhost.crt --import-key-path=/etc/ambari-server/certs/localhost.key --pem-password=hadoop --api-ssl-port=8443"
2) Configure trustedstore: "ambari-server setup-security --security-option=setup-truststore --truststore-reconfigure --truststore-type=jks --truststore-password=hadoop --truststore-path=/etc/ambari-server/certs/ambari-server-truststore"
Still looking into possibility of puppet-friendly setup.
