Automate HDP installation using Ambari Blueprints – Part 1

[#1]

What are Ambari Blueprints ?

Ambari Blueprints are definition of your HDP cluster in “JSON” format, it contents information about all the hosts in your cluster, their components, mapping of stack components with each hosts or hostgroups and other cool stuff. Using Blueprints we can call Ambari APIs to completely automate HDP installation process. Interesting stuff, isn’t it ?

Lets get started with single node cluster installation. Below are the steps to setup single-node HDP cluster with Ambari Blueprints.

.

Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications.

Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_reln...

.

Step 1: Install Ambari server using steps mentioned under below link

http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Installing_HDP_AMB/content/_download_the_...

.

Step 2: Register ambari-agent manually

Install ambari-agent package on all the nodes in the cluster and modify hostname to ambari server host(fqdn) in /etc/ambari-agent/conf/ambari-agent.ini

.

Step 3: Configure blueprints

Please follow below steps to create Blueprints

3.1 Create hostmapping.json file as shown below:

{
  "blueprint" : "single-node-hdp-cluster",
  "default_password" : "admin",
  "host_groups" :[
    {
      "name" : "host_group_1",
      "hosts" : [
        {
          "fqdn" : "<fqdn-of-single-node-cluster-machine>"
        }
      ]
    }
  ]
}

.

3.2 Create cluster_configuration.json file, it contents mapping of hosts to HDP components

{
  "configurations" : [ ],
  "host_groups" : [
    {
      "name" : "host_group_1",
      "components" : [
        {
          "name" : "NAMENODE"
        },
        {
          "name" : "SECONDARY_NAMENODE"
        },
        {
          "name" : "DATANODE"
        },
        {
          "name" : "HDFS_CLIENT"
        },
        {
          "name" : "RESOURCEMANAGER"
        },
        {
          "name" : "NODEMANAGER"
        },
        {
          "name" : "YARN_CLIENT"
        },
        {
          "name" : "HISTORYSERVER"
        },
        {
          "name" : "APP_TIMELINE_SERVER"
        },
        {
          "name" : "MAPREDUCE2_CLIENT"
        },
        {
          "name" : "ZOOKEEPER_SERVER"
        },
        {
          "name" : "ZOOKEEPER_CLIENT"
        }
      ],
      "cardinality" : "1"
    }
  ],
  "Blueprints" : {
    "blueprint_name" : "single-node-hdp-cluster",
    "stack_name" : "HDP",
    "stack_version" : "2.3"
  }
}

.

Step 4: Register blueprint with ambari server by executing below command

curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://<ambari-hostname>:8080/api/v1/blueprints/<blueprint-name>; -d @cluster_configuration.json

.

Srep 5: Pull the trigger! Below command will start cluster installation.

curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://<ambari-host>:8080/api/v1/clusters/<new-cluster-name>; -d @hostmapping.json

.

Step 6: We can track installation status by below REST call or we can check the same from ambari UI

curl -H "X-Requested-By: ambari" -X GET -u admin:admin http://<ambari-hostname>:8080/api/v1/clusters/mycluster/requests/

curl -H "X-Requested-By: ambari" -X GET -u admin:admin http://<ambari-hostname>:8080/api/v1/clusters/mycluster/requests/<request-number>;

.

Thank you for your time! Please read next part to see installation of HDP multinode cluster using Ambari Blueprints.

.

Happy Hadooping!! 🙂

[#2]

In Step 2, do remember to restart ambari-agent.

$ sudo ambari-agent restart

Regards,

Wendell

[#3]

Hi Kuldeep.

I actually did a restore from the backup and was able to preserve the cluster this way. I will keep the information you sent me in case I have a similar problem moving forward.

Thanks,

[#4]

@Kuldeep Kulkarni

Can you please elaborate step 4?
How do you created blueprint?
Where need to put this file "api/v1/blueprints" from this url "http://<ambari-hostname>:8080/api/v1/blueprints/<blueprint-name" ?

[#5]

I followed the steps given above but getting below error. 8080 port is not listening as ambari-server service is not running. I tried to start the service but it fails. Please suggest

curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://192.168.10.10:8080/api/v1/blueprints/single-node-hdp-cluster -d @cluster_configuration.json
curl: (7) Failed connect to 192.168.10.10:8080; Connection refused

[#6]

Hi Kuldeep

I'm having issues in setting up multi node ambari cluster using rest api. I will provide the screenshot very soon.But on the higher note the issue is that ambari UI doesn't register the new cluster and looks like its referencing the old clusters blueprint name

Let me know your thoughts

Thanks

Venkat

[#7]

Hello Kuldeep Kulkarni,

I have made all the step which you mentioned in Article, but HDP installation will take a long time after one-hour installation is still in processes.

Thanks

Ansari Faheem Ahmed

[#8]

Hi Kuldeep

I have ambari 2.6 local repositroy map as below but during installation, public repository being invoked by components and not using local repository which I have registered using Rest API. Please help me I am new to the ambari-blueprint

curl -i -H "X-Requested-By: ambari" -u admin:admin -X PUT \
-d'{ "Repositories" : {"repo_name":"HDP-2.5.0.0","base_url":"http://192.168.56.104/HDP/centos6", "verify_base_url":true }}' \ http://master.zdp.com:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP...


curl -i -H "X-Requested-By: ambari" -u admin:admin -X PUT \
-d'{ "Repositories" : {"repo_name":"HDP-UTILS-1.1.0.21","base_url":"http://192.168.56.104/hdpu", "verify_base_url":true }}' \ http://master.zdp.com:8080/api/v1/stacks/HDP/versions/2.5/operating_systems/redhat6/repositories/HDP...


[root@master aut]# more hostmapping.json
{
"blueprint" : "single-node-hdp-cluster",
"repository_version_id": 1,
"default_password" : "admin",
"host_groups" :[
{
"name" : "host_group_1",
"hosts" : [
{
"fqdn" : "master.zdp.com"
}
]
}
]
}

[root@master aut]# cat /etc/yum.repos.d/ambari-hdp-1.repo
[HDP-2.5-repo-1]
name=HDP-2.5-repo-1
baseurl=http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/2.5.0.0-1245

path=/
enabled=1
gpgcheck=0
[HDP-UTILS-1.1.0.21-repo-1]
name=HDP-UTILS-1.1.0.21-repo-1
baseurl=http://s3.amazonaws.com/dev.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6

path=/
enabled=1

[#9]

Hi @Kalyan Das ,

I think its better to create a new community thread for the issue you are facing .

it will be easily viewable by other community users for which they are willing to help .

Comments here wont allow posting screenshots and other things.