Reply
Expert Contributor
Posts: 66
Registered: ‎12-24-2015

cloudera-director update aws.reference.conf - Not able to update workers using director client.

[ Edited ]

we have upper limit for EC2 machines under d2.2x category but no such limit for i2.x EC2.

 

Here is what I did.

 

Started Cluster using director bootstrap command with just one worker of type d2.2x.

 

Now i am trying to update my cluster with 2 new worker of type i2.x EC2. I am still receving old error which i was getting with d2.2x.

 

com.cloudera.director.aws.shaded.com.amazonaws.AmazonServiceException: You have requested more instances (2) than your current instance limit of 1 allows for the specified instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit. (Service: AmazonEC2; Status Code: 400; Error Code: InstanceLimitExceeded; Request ID: c999a493-11e1-4b55-b07f-fdc717310943)
--------------------------------------------------

 

my conf is like below :

 

## exisitng worker

workers {
count: 1
minCount: 1

instance: ${instances.d22x} {

tags {
group: worker
}
}

roles {
HDFS: [DATANODE]
YARN: [NODEMANAGER]
HBASE: [REGIONSERVER]
}

}


#### my new workers
workers2 {
count: 3
minCount: 3

instance: ${instances.i2x} {

tags {
group: worker2
}
}

roles {
HDFS: [DATANODE]
YARN: [NODEMANAGER]
HBASE: [REGIONSERVER]
}

}

 

 

 

 

 

 

 

 

 

Highlighted
Cloudera Employee
Posts: 26
Registered: ‎09-21-2015

Re: cloudera-director update aws.reference.conf - Not able to update workers using director client.

Hi,

 

 

Since your conf file contains the old worker configs, it may be trying to add that a second time. Can you try with just the new worker groups?

 

Alternately you can also add a new group from the UI, in which case it won't impact the existing group.

Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Re: cloudera-director update aws.reference.conf - Not able to update workers using director client.

Hi Jayita,

 

Thanks for reply.

 

I have kept only new worker group and run update command. It did not work.

 

Are you saying me to add workers from cloudera manager UI, if you remember in this there is no director UI for me.

 

 

 

Cloudera Employee
Posts: 11
Registered: ‎09-21-2015

Re: cloudera-director update aws.reference.conf - Not able to update workers using director client.

The director ui should be running on port 7189 of your director host.

 

I'm wondering if the conf file you are supplying has the old worker group present, so director doesn't think it needs to delete the old workers group.  I'm going to try out a bootstrap remote command myself and try to give a concrete example of removing instances from a group, or changing the instances completely

Cloudera Employee
Posts: 11
Registered: ‎09-21-2015

Re: cloudera-director update aws.reference.conf - Not able to update workers using director client.

Ok, so, after testing this out, there is no way to use bootstrap-remote to modify a cluster.  You have to either use the UI or the API (python/java client) 

 

 

Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Re: cloudera-director update aws.reference.conf - Not able to update workers using director client.

So u mean update command from director client can't add workers ?
Cloudera Employee
Posts: 11
Registered: ‎09-21-2015

Re: cloudera-director update aws.reference.conf - Not able to update workers using director client.

Bootstrap remote is used mainly to get a cluster up and running.  Once its running, if you run a second bootstrap remote, it won't modify the existing cluster groups.  That includes adding/removing groups.

 

You need to use the api to modify the cluster

Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Re: cloudera-director update aws.reference.conf - Not able to update workers using director client.

I didn't use "bootstrap remote" command to build cluster. I used simple "bootstrap" followed by "update" command.

Where to look for API
Cloudera Employee
Posts: 11
Registered: ‎09-21-2015

Re: cloudera-director update aws.reference.conf - Not able to update workers using director client.

I see.  I just looked at the documentation and it looks like bootstrap-remote is used to get an existing Director server to create a cluster.  

http://www.cloudera.com/documentation/director/latest/topics/director_cli_commands.html

 

Bootstrap and update are used to make changes to an existing cloudera manager.

 

So we are talking about different things.  Can you confirm what your setup is?

Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Re: cloudera-director update aws.reference.conf - Not able to update workers using director client.

I used only "bootstrap" command to Start cluster using director client.
Now trying to expand cluster using "update" command.