Automate HDP installation using Ambari Blueprints – Part 2

[#1]

In previous post we have seen how to install single node HDP cluster using Ambari Blueprints. In this post we will see how to Automate HDP installation using Ambari Blueprints.

.

Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications.

Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_reln...

.

Below are simple steps to install HDP multinode cluster using internal repository via Ambari Blueprints.

.

Step 1: Install Ambari server using steps mentioned under below link

http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Installing_HDP_AMB/content/_download_the_...

.

Step 2: Register ambari-agent manually

Install ambari-agent package on all the nodes in the cluster and modify hostname to ambari server host(fqdn) in /etc/ambari-agent/conf/ambari-agent.ini

.

Step 3: Configure blueprints

Please follow below steps to create Blueprints

.

3.1 Create hostmapping.json file as shown below:

Note – This file will have information related to all the hosts which are part of your HDP cluster.

{
"blueprint" : "multinode-hdp",
"default_password" : "hadoop",
"host_groups" :[
  {
  "name" : "host2",
  "hosts" : [
  {
  "fqdn" : "host2.crazyadmins.com"
  }
  ]
  },
  {
  "name" : "host3",
  "hosts" : [
  {
  "fqdn" : "host3.crazyadmins.com"
  }
  ]
  },
  {
  "name" : "host4",
  "hosts" : [
  {
  "fqdn" : "host4.crazyadmins.com"
  }
  ]
  }
]
}

.

3.2 Create cluster_configuration.json file, it contents mapping of hosts to HDP components

{
 "configurations": [],
 "host_groups": [{
 "name": "host2",
 "components": [{
 "name": "PIG"
 }, {
 "name": "METRICS_COLLECTOR"
 }, {
 "name": "KAFKA_BROKER"
 }, {
 "name": "HISTORYSERVER"
 }, {
 "name": "HBASE_REGIONSERVER"
 }, {
 "name": "OOZIE_CLIENT"
 }, {
 "name": "HBASE_CLIENT"
 }, {
 "name": "NAMENODE"
 }, {
 "name": "SUPERVISOR"
 }, {
 "name": "HCAT"
 }, {
 "name": "METRICS_MONITOR"
 }, {
 "name": "APP_TIMELINE_SERVER"
 }, {
 "name": "NODEMANAGER"
 }, {
 "name": "HDFS_CLIENT"
 }, {
 "name": "HIVE_CLIENT"
 }, {
 "name": "FLUME_HANDLER"
 }, {
 "name": "DATANODE"
 }, {
 "name": "WEBHCAT_SERVER"
 }, {
 "name": "ZOOKEEPER_CLIENT"
 }, {
 "name": "ZOOKEEPER_SERVER"
 }, {
 "name": "STORM_UI_SERVER"
 }, {
 "name": "HIVE_SERVER"
 }, {
 "name": "FALCON_CLIENT"
 }, {
 "name": "TEZ_CLIENT"
 }, {
 "name": "HIVE_METASTORE"
 }, {
 "name": "SQOOP"
 }, {
 "name": "YARN_CLIENT"
 }, {
 "name": "MAPREDUCE2_CLIENT"
 }, {
 "name": "NIMBUS"
 }, {
 "name": "DRPC_SERVER"
 }],
 "cardinality": "1"
 }, {
 "name": "host3",
 "components": [{
 "name": "ZOOKEEPER_SERVER"
 }, {
 "name": "OOZIE_SERVER"
 }, {
 "name": "SECONDARY_NAMENODE"
 }, {
 "name": "FALCON_SERVER"
 }, {
 "name": "ZOOKEEPER_CLIENT"
 }, {
 "name": "PIG"
 }, {
 "name": "KAFKA_BROKER"
 }, {
 "name": "OOZIE_CLIENT"
 }, {
 "name": "HBASE_REGIONSERVER"
 }, {
 "name": "HBASE_CLIENT"
 }, {
 "name": "HCAT"
 }, {
 "name": "METRICS_MONITOR"
 }, {
 "name": "FALCON_CLIENT"
 }, {
 "name": "TEZ_CLIENT"
 }, {
 "name": "SQOOP"
 }, {
 "name": "HIVE_CLIENT"
 }, {
 "name": "HDFS_CLIENT"
 }, {
 "name": "NODEMANAGER"
 }, {
 "name": "YARN_CLIENT"
 }, {
 "name": "MAPREDUCE2_CLIENT"
 }, {
 "name": "DATANODE"
 }],
 "cardinality": "1"
 }, {
 "name": "host4",
 "components": [{
 "name": "ZOOKEEPER_SERVER"
 }, {
 "name": "ZOOKEEPER_CLIENT"
 }, {
 "name": "PIG"
 }, {
 "name": "KAFKA_BROKER"
 }, {
 "name": "OOZIE_CLIENT"
 }, {
 "name": "HBASE_MASTER"
 }, {
 "name": "HBASE_REGIONSERVER"
 }, {
 "name": "HBASE_CLIENT"
 }, {
 "name": "HCAT"
 }, {
 "name": "RESOURCEMANAGER"
 }, {
 "name": "METRICS_MONITOR"
 }, {
 "name": "FALCON_CLIENT"
 }, {
 "name": "TEZ_CLIENT"
 }, {
 "name": "SQOOP"
 }, {
 "name": "HIVE_CLIENT"
 }, {
 "name": "HDFS_CLIENT"
 }, {
 "name": "NODEMANAGER"
 }, {
 "name": "YARN_CLIENT"
 }, {
 "name": "MAPREDUCE2_CLIENT"
 }, {
 "name": "DATANODE"
 }],
 "cardinality": "1"
 }],
 "Blueprints": {
 "blueprint_name": "multinode-hdp",
 "stack_name": "HDP",
 "stack_version": "2.3"
 }
}

.

Step 4: Create an internal repository map

.

4.1: hdp repository – copy below contents, modify base_url to add hostname/ip-address of your internal repository server and save it in repo.json file.

{
"Repositories" : {
  "base_url" : "http://<ip-address-of-repo-server>/hdp/centos6/HDP-2.3.4.0",
  "verify_base_url" : true
}
}

.

4.2: hdp-utils repository – copy below contents, modify base_url to add hostname/ip-address of your internal repository server and save it in hdputils-repo.json file.

{
"Repositories" : {
  "base_url" : "http://<ip-address-of-repo-server>/hdp/centos6/HDP-UTILS-1.1.0.20",
  "verify_base_url" : true
}
}

.

Step 5: Register blueprint with ambari server by executing below command

curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://<ambari-server-hostname>:8080/api/v1/blueprints/multinode-hdp -d @cluster_config.json

.

Step 6: Setup Internal repo via REST API.

Execute below curl calls to setup internal repositories.

curl -H "X-Requested-By: ambari" -X PUT -u admin:admin http://<ambari-server-hostname>:8080/api/v1/stacks/HDP/versions/2.3/operating_systems/redhat6/reposi... -d @repo.json

curl -H "X-Requested-By: ambari" -X PUT -u admin:admin http://<ambari-server-hostname>:8080/api/v1/stacks/HDP/versions/2.3/operating_systems/redhat6/reposi... -d @hdputils-repo.json

.

Step 7: Pull the trigger! Below command will start cluster installation.

curl -H "X-Requested-By: ambari" -X POST -u admin:admin http://<ambari-server-hostname>:8080/api/v1/clusters/multinode-hdp -d @hostmap.json

.

Note - Please refer third part of this tutorial if you want to setup a multinode cluster with Namenode HA

.

Please feel free to comment if you need any further help on this. Happy Hadooping!!

[#2]

Hi @Kuldeep Kulkarni ,

thanks for your helpful posts. Is it planned to continue this series.....I am looking forward to a Blueprint including Namenode-/HMaster-HA as well as Kerberos

Thanks in advance...

[#3]

@Gerd Koenig

Thanks. Sure In next post, We can cover NN/RM HA via Blueprints.

[#4]

@Gerd Koenig - Next tutorial is up - Blueprint setup with NN HA - https://community.hortonworks.com/articles/61358/automate-hdp-installation-using-ambari-blueprints-2...

[#5]

Hi, Kuldeep Kulkarni. I am trying to create cluster using ambari blueprint. But after registering blueprint and triggering cluster creation request successfully when I see progress in Ambari UI it stays stuck in PENDING HOST ASSIGNMENT which means that it is not able to connect or find hosts. But I have checked all the fqdn and ambari agents are also up and running. My approach is to store/ copy ambari.repo and hdp.repo in all hosts before hand so I have skipped step 4. What could be the problem???...please help!!!

[#6]

Hi Kuldeep

Do you have any post for ambari installation using blueprint

[#7]

Hi @Kuldeep Kulkarni

Does this tutorial still work with Ambari 2.6?