Cluster Scaling via Director API

[#1]

I understand that scaling the cluster is done by redefining the cluster itself.  What I don't know (and have been unable to find) is how much information do I need to include in PUT to change the cluster.  There's a lot of information that is generated automatically (IP addresses for example) that I don't have/want to include, I'd rather just say I want X nodes of Y template.  Anyone have a resource I can look at?  I've spent sometime working in the API console and the documentation just seems lacking here...

[#2 · Accepted solution]

It's good to hear that you're looking into this. I don't think we have a lot of API users at the moment, so your feedback will be very useful to us.

So you'll really need PUT for the cluster template in order to make an update rather than the cluster object. The cluster object is a bit more ephemeral in nature and reflects the current active state of the cluster.

Are you using one of the SDKs (https://github.com/cloudera/director-sdk, only available in Java/Python)? It sounds like you're doing something a little more manual, but I'd encourage you to use those if possible. Otherwise, the API console is probably your best bet for documentation at the moment. Assuming Director is running on your local machine, you'd navigate to:

http://localhost:7189/api-console

Let me know if that points you in the right direction.

[#3]

My original thought was to talk JSON to the API directly. The API console is where I've been spending my time trying to understand what's needed to resize the cluster. I'd glanced at the sdk, but looking at it further, it might work for what I'm trying to do (faux EMR).

[#4]

FYI - Further investigation in to the python version of the SDK looks like it will do what I need. Thanks for pointing me back to it.

[#5]

@Mike Wilson : Hello ,

Can you please provide a high level over view i should follow for the below requirement.

Requirement: We want to develop a script which will perform instance addition, master service migration,datanode replacement.

On our cluster we have Director 2.6 and use AWS as infra.

Does cloudera director provide api's to perform the above service .I guess the above requirement will need a mix of director and manager api's.

Is ansible suitable for the above requirement.

[#6]

Tauqeer,

Cloudera Director's API will provide what you need to add instances and replace data nodes. Master migration is a little trickier. Cloudera Director can help with creating instances for the migration, but you will have to perform the migration in Cloudera Manager. See https://www.cloudera.com/documentation/director/latest/topics/director_create_ha_clusters.html#migra... for information on how to use Cloudera Director when migrating masters.

These tasks will mainly use Cloudera Director's "update" API. See https://github.com/cloudera/director-sdk/blob/master/java-client/src/main/java/com/cloudera/director... or https://github.com/cloudera/director-sdk/blob/master/python-client/cloudera/director/latest/Clusters...

There are examples in both java and python to get you started.

https://github.com/cloudera/director-sdk

[#7]

I have the beginning of Flask based REST api I was working on before my project got discontinued. I tried my best to follow the Director API routes. I'll upload it to Github and share the link here.