Support Questions
Find answers, ask questions, and share your expertise

Automating ambari/hdp deployment using REST API (python client)

Automating ambari/hdp deployment using REST API (python client)

New Contributor

But I'm using python_client for that. I've found that there's no concrete or good documentation for python client of ambari_api. Can someone redirect me to some good resource for python_client of ambari_api?

Or

Can someone post a test snippet here. Consider this. I have to create HDFS service with its components (NN, SNN, DN) and then create all the configs (default) and then start that.

7 REPLIES 7

Re: Automating ambari/hdp deployment using REST API (python client)

Expert Contributor

You are right that there doesn't seem to be any recent work on the Python client for Ambari - you are free to try that approach but may I suggest some alternatives to automating your HDP deployment:

Re: Automating ambari/hdp deployment using REST API (python client)

New Contributor

Hi @anarasimham first of all thanks for the detailed answer, appreciate that.

Actually I'm automating all of this using chef. I can't use any other tool.

I want a behavior like Cloudera-API where we can just create services and roles on any host that we want dynamically with the default configuration or custom configurations. So can anything like that be done with AMbari APi?

I've already automated the prereqs like ambari server and agents. It's the HDP automation that I want to do. Here is what i want to do using API calls.

1) Create a service (i.e HDFS)

2) Create a component on given node (NN), with default configuration

3) create a component on given node (SNN) with default configuration

4) create a component on given node (DN 1) with default configuration

5) create a component on given node (DN 2) with default configuration

6) create other components if required.

7) Start a service

Note: Can you also elaborate how can we pass default configs, because when I created the components, they got installed with empty configs.

Thanks much.

Re: Automating ambari/hdp deployment using REST API (python client)

New Contributor

Hi @anarasimham first of all thanks for the detailed answer, appreciate that.

Actually I'm automating all of this using chef. I can't use any other tool.

I want a behavior like Cloudera-API where we can just create services and roles on any host that we want dynamically with the default configuration or custom configurations. So can anything like that be done with AMbari APi?

I've already automated the prereqs like ambari server and agents. It's the HDP automation that I want to do. Here is what i want to do using API calls.

1) Create a service (i.e HDFS)

2) Create a component on given node (NN), with default configuration

3) create a component on given node (SNN) with default configuration

4) create a component on given node (DN 1) with default configuration

5) create a component on given node (DN 2) with default configuration

6) create other components if required.

7) Start a service

Note: Can you also elaborate how can we pass default configs, because when I created the components, they got installed with empty configs.

Thanks much.

Re: Automating ambari/hdp deployment using REST API (python client)

Super Mentor

@Shahroz Haider

As you want to automate the Ambari HDP cluster setup and some additional configuration as well. In that case we will suggest you to use "Ambari Blueprint" concept which can help in creating and configuring Ambari Cluster in just 2-3 steps using simple JSON files by making the Ambari API calls.

The advantage of using ambari blueprint is that you can easily integrate it with any available option like Using "Curl" you can invoke the Blueprint registration and cluster deployment... Similarly you can invoke the same Rest API calls using Python Clients as well.

To know more about Ambari Blueprints please refer to: https://cwiki.apache.org/confluence/display/AMBARI/Blueprints

A detailed example of 6-7 Set of articles from basic Ambari Blueprint to Advance configuration can be found in the following HCC Article:

Basic Blueprint Single Node

Part-1). https://community.hortonworks.com/articles/47170/automate-hdp-installation-using-ambari-blueprints.h...

Configuring NameNode HA Multinode cluster
Part-2). https://community.hortonworks.com/content/kbentry/47171/automate-hdp-installation-using-ambari-bluep...

Part-3). https://community.hortonworks.com/articles/61358/automate-hdp-installation-using-ambari-blueprints-2...

Part-4). https://community.hortonworks.com/articles/70189/automate-hdp-installation-using-ambari-blueprints-3...

Secure Cluster
Part-5). https://community.hortonworks.com/articles/78969/automate-hdp-installation-using-ambari-blueprints-4...

Rest of the parts links you can get in the same article.

.

Re: Automating ambari/hdp deployment using REST API (python client)

New Contributor

Thanks @Jay Kumar SenSharma. But as per my understanding blueprints make this whole process static. As I'm using chef. so what i need is a dynamic behavior. By dynamic I mean, i should be able to say which components are installed on which nodes.

1) Lets say, I have 2 nodes. Can i create a first blueprint (along with its configurations) of NameNode for first node. And then for second node I create a second blueprint (along with configurations) for DaraNode.?

2) Also as you know that there are huge number of config properties. In hadoop only we can have core-site and hdfs-site. So how can I pass the default configs in blueprint (like Ambari UI provides recommended configs) rather than creating each from scratch?

Thanks

Re: Automating ambari/hdp deployment using REST API (python client)

Super Mentor

@Shahroz Haider

1. But as per my understanding blueprints make this whole process static.

>>> No, Blueprint is just a Template (JSON) of the cluster formation. The actual host mapping remains in a separate JSON file. So you can add as many hosts as part of your blueprint.

.

2. Can i create a first blueprint (along with its configurations) of NameNode for first node. And then for second node I create a second blueprint (along with configurations) for DaraNode.?

>>> The advantage of blueprint is that you do not need to create multiple blueprints for the same cluster. In one shot you can create the whole cluster (multiple blueprints are not needed).

Blueprint works on "HOST_GROUPS" concept. So you can define multiple host_groups inside the Host_mapping JSON file to tell which node should have DataNode, which nodes should have NodeManagers ...etc

.

3. Also as you know that there are huge number of config properties. In hadoop only we can have core-site and hdfs-site. So how can I pass the default configs in blueprint

>>> The advantage of blueprint is that it allows you to define the additional configuration properties inside the "blueprint.json" file for your conponents/services. However if you do not explicitly define all those properties then it will use the component's default properties on it's own.

.

Re: Automating ambari/hdp deployment using REST API (python client)

New Contributor

@Jay Kumar SenSharma

BTW thanks a lot for clearing that up.

1) I'm actually using python_client of API rather than using curl or writing python wrapper on http calls. So can this whole automation with blueprints can be done with ambari_api python_client?

2) I got your point of deploying cluster in oneshot using blueprint. But my use case is different. Here's what I have to do. For HDFS i have to write 4 isolated python services. Following should be the expected behavior (Assume that ambari-server and ambari-agents are installed. Also assume that we have a two node cluster named node1 and node2)

i) Create service HDFS with all default configs

ii) Create component Namenode with all default configs (on node1)

iii) Create component Datanode with all default configs (on node2)

iv) Start service HDFS

Now can you go through this and tell me how will i create 4 different blueprints for these four aforementioned services, as we cant have just one blueprint because all 4 services are isolated. All services would just connect with Ambari server and do their things according to credentials.

Thanks