Unattended/Fully automated deployment of a cluster with a Json file

[#1]

Hello Team,

Currently I am trying to fully automate a deployment of a trivial cluster (one node cluster) by using Ansible. I found out that I have to use the cloudera-manager-installer binary and pass to it the following arguments : 

--i-agree-to-all-licenses 
--noprompt

--noreadme

--nooptions

Actually it is the same like installing the following packages :

 - cloudera-manager-daemons

 - cloudera-manager-server

 - cloudera-manager-server-db-2

From that point when you open your browser on port 7180, you will have to login and then follow an initial wizard. My question is how I can skip or automate the steps on this wizard and finally import a json file in order to deploy my cluster. Thanks in advance.

[#2]

The CM Api is available at that point. Try the /cm/importClusterTemplate node. You may need to create a empty cluster first with /clusters. To go through the wizard try : /api/v12/clusters/{clusterName}/commands/firstRun.

https://cloudera.github.io/cm_api/apidocs/v12/

[#3]

Thanks for your quick response mbigelow. I will try your suggestions and I will come back with my results .

[#4]

I am close to achieve what I wanted (will make a post about the steps). I was wondering what is the difference between importing a cluster template and doing a full deployment by using : /api/v12/cm/deployment. From my understanding the cluster template is the blueprint of the cluster and then from that point you can apply different deployments on this blueprint? Briefly what I did so far :

1) Start the Cloudera-scm-server-db

2) Start the Cloudera-scm-server

3) Start the Cloudera-scm-agent

4) Import a Template from a newly built cluster

What I wanted to know if its better to make the configuration changes that I want on my cluster and then instead of importing the cluster template to use the deployment API Call. Thanks in advance.

[#5]

I don't know if I can say for certain without testing them out.  I feel like /cm/deployment has always been there while the importClusterTemplate was added when they added blueprints.  Both sound like they will take in a json object that describes a fully cluster.  The deployment node will fail if any portion of it exists and rollback all changes.  I don't know if the templates are more flexible or contain conditions.

[#6]

Thanks for your quick reply. I will try it and let you know about the results.

[#7]

Does anybody know how I can add the cloudera management services through an API call? Is it possible?

[#8 · Accepted solution]

I finally managed to fully automate the deployment of a Cluster by using just API calls. Below you can find the steps:

1) Enable CDM and CDH repository

2) Install Java

3) Install the following packages : 

    - cloudera-manager-agent

    - cloudera-manager-daemons

    - cloudera-manager-server

    - cloudera-manager-server-db-2

4) If you want to install the cluster through parcels you don't need to install anything else. I have chosen to go with packages, so I had to install : 

    - hadoop

    - hadoop-hdfs

    - hadoop-libhdfs

    - hadoop-yarn

    - hadoop-mapreduce

    - hadoop-client

    - openssl

5) Start the Cloudera database 

6) Start the Cloudera Server

7) Start the Cloudera agent

😎 Enable the trial version of the Enterprise Hub edition. This version gives you the ability to import templates.

9) Import the template of the cluster

The question is how does someone can know how its template should look. The answer in this question is to build a testing cluster manually, configure it as you want and then export its configuration. I hope this helps. Thanks again for your help.

[#9]

Do you have a documentation for this