Deploying multinode HDP with puppet

[#1]

Dear colleagues,

I am trying to deploy multinode HDP with puppet, but have some troubles with services preconfiguration.

Regarding ambari install and machines configuration everything is understandable:

1) Setup ssh, libraries, shares etc. using puppet.

2) Install ambari, also puppet - https://forge.puppetlabs.com/vzach/ambari

However the problem appears when configuring services. For example, hbase, MR, ozzie, etc. For this I had not found any puppet libraries. I`ve seen that there is possible to manage a cluster via blueprints or ambari shell. Also had found that there is some limited support of bigtop. The bigtop seems to be more capable, but had not found any description of using it with ambari.

Would be very grateful for some help this question.

[#2]

I would strongly recommend the use of Ambari to manage your HDP clusters.

If you need to automate the installation of multiple HDP clusters, Ambari blueprints are a very nice feature to reduce the cost of re-installation.

[#3]

I`ve found only curl or some custom app for deploying configuration via blueprints. Is there any other way of deploying it?

[#4]

As Josh already mentioned, the best way to deploy HDP clusters is Ambari. Bigtop is a packaging project that has some ability to deploy unmanaged cluster for testing purposes. Ambari on the other hand is the cluster managing software. It may come that one day we will see Ambari as one of the packages in Bigtop, but even in this case the deployment of cluster will be done by Ambari blueprints.

[#5 · Accepted solution]

@Vladislav Falfushinsky

In addition to answer given by @Josh Elser. I would like to share my tutorials on Ambari Blueprints to help you get started quickly :

How to install single node HDP cluster using Ambari blueprint:

https://community.hortonworks.com/content/kbentry/47170/automate-hdp-installation-using-ambari-bluep...

How to install multi node HDP cluster using Ambari blueprint:

https://community.hortonworks.com/content/kbentry/47171/automate-hdp-installation-using-ambari-bluep...

How to install multi node HDP cluster with Namenode HA using Ambari blueprint:

https://community.hortonworks.com/articles/61358/automate-hdp-installation-using-ambari-blueprints-2...

How to install single node HDP cluster with Kerberos authentication using Ambari blueprint:

https://community.hortonworks.com/articles/70189/automate-hdp-installation-using-ambari-blueprints-3...

How to install multinode HDP cluster with Kerberos authentication using Ambari blueprint:

https://community.hortonworks.com/articles/78969/automate-hdp-installation-using-ambari-blueprints-4...

[#6]

@Kuldeep Kulkarni

Many thanks. I had already seem them. Is there any cli rather then curl utility for deploying blueprints?

[#7]

I would follow Josh's recommendation to manage the cluster once ambari is installed. Utilizing a configuration management tool like puppet is still useful to do all the preconfigurations before ambari is utilized.

[#8]

Faced several issues when making the deployment:

1. Ambari puppet configurations that can be found on puppetforge are tested only for CentOS and still some features are not supported. I am using debian. Trying to test and make some rewrites for debian and fix some issues.

2. It would be nice to have some utility for managing blueprints (something similar to StackMaster for managing Amazon CloudFormation stacks). Had been thinking about such utility, but it will happen after first point.

[#9]

Faced some question of consistency. Is this configuration https://forge.puppet.com/unibet/ambari fully consistent with latest ambari version?

[#10]

Completed with some basic puppet setup. It required rewriting of some code from https://forge.puppet.com/unibet/ambari. Also found out that following puppet configuration does not support ssl. Trying to implement it, but had not found any manual how to configure ssl for ambari not using "ambari-server setup-security" as specified here https://community.hortonworks.com/articles/39865/enabling-https-for-ambariserver-and-troubleshootin..... I assume that only options needed to make ssl working are:

1) Keystore for ambari services:

"

ssl.trustStore.path=<path-to-keystore-file>

ssl.trustStore.type=jks

ssl.trustStore.password=<keystore-password-here>

"

2) HTTPS options:

"

client.api.ssl.port=8443

api.ssl=true

client.api.ssl.cert_name=https.crt

client.api.ssl.key_name=https.key

"

Isn`t it simpler to use nginx with ssl in front of ambari to provide https?

Just tried to make a manual setup with generating certificates and filling the needed fields in /etc/ambari-server/conf/ambari.properties, afterwards started "ambari-server setup -s", but got "ambari-server restart" failed with error about problem with certificates and truststore. Seems it makes some transformations in process of running setup.

Found out that there are some options for ambari-server setup-security option:

1) Configure https: "ambari-server setup-security --security-option=setup-https --api-ssl=true --import-cert-path=/etc/ambari-server/certs/localhost.crt --import-key-path=/etc/ambari-server/certs/localhost.key --pem-password=hadoop --api-ssl-port=8443"

2) Configure trustedstore: "ambari-server setup-security --security-option=setup-truststore --truststore-reconfigure --truststore-type=jks --truststore-password=hadoop --truststore-path=/etc/ambari-server/certs/ambari-server-truststore"

Still looking into possibility of puppet-friendly setup.