Automating ambari/hdp deployment using REST API (python client)

[#1]

But I'm using python_client for that. I've found that there's no concrete or good documentation for python client of ambari_api. Can someone redirect me to some good resource for python_client of ambari_api?

Or

Can someone post a test snippet here. Consider this. I have to create HDFS service with its components (NN, SNN, DN) and then create all the configs (default) and then start that.

[#2]

You are right that there doesn't seem to be any recent work on the Python client for Ambari - you are free to try that approach but may I suggest some alternatives to automating your HDP deployment:

• Cloudbreak - this integrates fully with AWS/Azure/GCP/OpenStack, so will work well if you are attempting installation on one of those platforms. http://hortonworks.github.io/cloudbreak-docs/latest/
• Use the Ambari Blueprints functionality to setup a cluster. Note that this requires the little bit of pre-work that is installing Ambari agents on all nodes in the cluster. https://community.hortonworks.com/articles/47170/automate-hdp-installation-using-ambari-blueprints.h...
• Use the Ambari REST API through cURL. Detailed documentation exists for this and should allow you to do everything you need. The following page details how to install the Ambari agent on all hosts, but please browse through the rest of those links on the sidebar to get an idea of what it can do. https://cwiki.apache.org/confluence/display/AMBARI/Installing+ambari-agent+on+target+hosts.

[#3]

Hi @anarasimham first of all thanks for the detailed answer, appreciate that.

Actually I'm automating all of this using chef. I can't use any other tool.

I want a behavior like Cloudera-API where we can just create services and roles on any host that we want dynamically with the default configuration or custom configurations. So can anything like that be done with AMbari APi?

I've already automated the prereqs like ambari server and agents. It's the HDP automation that I want to do. Here is what i want to do using API calls.

1) Create a service (i.e HDFS)

2) Create a component on given node (NN), with default configuration

3) create a component on given node (SNN) with default configuration

4) create a component on given node (DN 1) with default configuration

5) create a component on given node (DN 2) with default configuration

6) create other components if required.

7) Start a service

Note: Can you also elaborate how can we pass default configs, because when I created the components, they got installed with empty configs.

Thanks much.

[#4]

Hi @anarasimham first of all thanks for the detailed answer, appreciate that.

Actually I'm automating all of this using chef. I can't use any other tool.

I want a behavior like Cloudera-API where we can just create services and roles on any host that we want dynamically with the default configuration or custom configurations. So can anything like that be done with AMbari APi?

I've already automated the prereqs like ambari server and agents. It's the HDP automation that I want to do. Here is what i want to do using API calls.

1) Create a service (i.e HDFS)

2) Create a component on given node (NN), with default configuration

3) create a component on given node (SNN) with default configuration

4) create a component on given node (DN 1) with default configuration

5) create a component on given node (DN 2) with default configuration

6) create other components if required.

7) Start a service

Note: Can you also elaborate how can we pass default configs, because when I created the components, they got installed with empty configs.

Thanks much.

[#5]

@Shahroz Haider

As you want to automate the Ambari HDP cluster setup and some additional configuration as well. In that case we will suggest you to use "Ambari Blueprint" concept which can help in creating and configuring Ambari Cluster in just 2-3 steps using simple JSON files by making the Ambari API calls.

The advantage of using ambari blueprint is that you can easily integrate it with any available option like Using "Curl" you can invoke the Blueprint registration and cluster deployment... Similarly you can invoke the same Rest API calls using Python Clients as well.

To know more about Ambari Blueprints please refer to: https://cwiki.apache.org/confluence/display/AMBARI/Blueprints

A detailed example of 6-7 Set of articles from basic Ambari Blueprint to Advance configuration can be found in the following HCC Article:

Basic Blueprint Single Node

Part-1). https://community.hortonworks.com/articles/47170/automate-hdp-installation-using-ambari-blueprints.h...

Configuring NameNode HA Multinode cluster
Part-2). https://community.hortonworks.com/content/kbentry/47171/automate-hdp-installation-using-ambari-bluep...

Part-3). https://community.hortonworks.com/articles/61358/automate-hdp-installation-using-ambari-blueprints-2...

Part-4). https://community.hortonworks.com/articles/70189/automate-hdp-installation-using-ambari-blueprints-3...

Secure Cluster
Part-5). https://community.hortonworks.com/articles/78969/automate-hdp-installation-using-ambari-blueprints-4...

Rest of the parts links you can get in the same article.

.

[#6]

Thanks @Jay Kumar SenSharma. But as per my understanding blueprints make this whole process static. As I'm using chef. so what i need is a dynamic behavior. By dynamic I mean, i should be able to say which components are installed on which nodes.

1) Lets say, I have 2 nodes. Can i create a first blueprint (along with its configurations) of NameNode for first node. And then for second node I create a second blueprint (along with configurations) for DaraNode.?

2) Also as you know that there are huge number of config properties. In hadoop only we can have core-site and hdfs-site. So how can I pass the default configs in blueprint (like Ambari UI provides recommended configs) rather than creating each from scratch?

Thanks

[#7]

@Shahroz Haider

1. But as per my understanding blueprints make this whole process static.

>>> No, Blueprint is just a Template (JSON) of the cluster formation. The actual host mapping remains in a separate JSON file. So you can add as many hosts as part of your blueprint.

.

2. Can i create a first blueprint (along with its configurations) of NameNode for first node. And then for second node I create a second blueprint (along with configurations) for DaraNode.?

>>> The advantage of blueprint is that you do not need to create multiple blueprints for the same cluster. In one shot you can create the whole cluster (multiple blueprints are not needed).

Blueprint works on "HOST_GROUPS" concept. So you can define multiple host_groups inside the Host_mapping JSON file to tell which node should have DataNode, which nodes should have NodeManagers ...etc

.

3. Also as you know that there are huge number of config properties. In hadoop only we can have core-site and hdfs-site. So how can I pass the default configs in blueprint

>>> The advantage of blueprint is that it allows you to define the additional configuration properties inside the "blueprint.json" file for your conponents/services. However if you do not explicitly define all those properties then it will use the component's default properties on it's own.

.

[#8]

@Jay Kumar SenSharma

BTW thanks a lot for clearing that up.

1) I'm actually using python_client of API rather than using curl or writing python wrapper on http calls. So can this whole automation with blueprints can be done with ambari_api python_client?

2) I got your point of deploying cluster in oneshot using blueprint. But my use case is different. Here's what I have to do. For HDFS i have to write 4 isolated python services. Following should be the expected behavior (Assume that ambari-server and ambari-agents are installed. Also assume that we have a two node cluster named node1 and node2)

i) Create service HDFS with all default configs

ii) Create component Namenode with all default configs (on node1)

iii) Create component Datanode with all default configs (on node2)

iv) Start service HDFS

Now can you go through this and tell me how will i create 4 different blueprints for these four aforementioned services, as we cant have just one blueprint because all 4 services are isolated. All services would just connect with Ambari server and do their things according to credentials.

Thanks