Unable to update existing cluster using the same credentials that created it

[#1]

I used cloudera-director-1.0.0 (the CLI) to create a CDH cluster, using an aws.conf file based off of aws.simple.conf. Now, to verify that I am able to expand the cluster if needed, I modified the conf file to change the number of data nodes from 8 to 9. I then ran ./bin/cloudera-director update aws.simple.conf. However, I get back this:

Process logs can be found at /home/ec2-user/cloudera/cloudera-director-1.0.0/logs/application.log
Cloudera Director 1.0.0 initializing ...
Updating environment state ...done
Unexpected internal error (see logs): null

If I look at the application.log, the final (seemingly relevant bit) is:

[2014-11-11 15:45:36] INFO [main] - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating parcels and version compatibility
[2014-11-11 15:45:40] ERROR [main] - com.cloudera.launchpad.Application: Command execution failed
javax.ws.rs.NotAuthorizedException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_45]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_45]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_45]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_45]
at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:452) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.checkResponse(ClientProxyImpl.java:298) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:586) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:552) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at com.sun.proxy.$Proxy131.getConfig(Unknown Source) ~[na:na]
at com.cloudera.launchpad.bootstrap.validators.GenericClusterTemplateValidator.validate(GenericClusterTemplateValidator.java:82) ~[launchpad-bootstrap-1.0.0.jar!/:1.0.0]
at com.cloudera.launchpad.model.ValidatorRegistry.validate(ValidatorRegistry.java:82) ~[launchpad-model-1.0.0.jar!/:1.0.0]
at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1.validate(ClustersResourceV1.java:574) ~[launchpad-api-resources-1.0.0.jar!/:1.0.0]
at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1.update(ClustersResourceV1.java:304) ~[launchpad-api-resources-1.0.0.jar!/:1.0.0]
at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1.update(ClustersResourceV1.java:260) ~[launchpad-api-resources-1.0.0.jar!/:1.0.0]
at com.cloudera.launchpad.commands.UpdateCommand.run(UpdateCommand.java:106) ~[launchpad-cli-1.0.0.jar!/:1.0.0]
at com.cloudera.launchpad.Application.run(Application.java:111) [launchpad-cli-1.0.0.jar!/:1.0.0]
at org.springframework.boot.SpringApplication.runCommandLineRunners(SpringApplication.java:677) [spring-boot-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:695) [spring-boot-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:321) [spring-boot-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
at com.cloudera.launchpad.Application.start(Application.java:74) [launchpad-cli-1.0.0.jar!/:1.0.0]
at com.cloudera.launchpad.Application.main(Application.java:43) [launchpad-cli-1.0.0.jar!/:1.0.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_45]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_45]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:53) [launchpad-cli-1.0.0.jar!/:1.0.0]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

The part that jumped out at me is javax.ws.rs.NotAuthorizedException, of course. But, as I said, I used this exact same conf file with the same accessKeyId and secretAccessKey to create the cluster in the first place. I can run './bin/cloudera-director status aws.simple.conf' and get back a full listing of all of the instances that were spun up. However, I know that the credentials I'm using are *not* for a so-called "root" AWS account. There are various things that I'm not allowed to do, like list/create/update users (iAM), or various billing things. Is it possible that there is something (somehow) that makes a call to one of those functions in the 'update' that is not called in the 'bootstrap' code? Well, I'm probably overstepping my bounds as a problem reporter. Let me know if there's anything else I can provide in the way of details or things I can try.

[#2]

Hi David - 

Have you changed the default Cloudera Manager password from admin/admin to somethind else? That may be the source of this error. 

[#3]

That must be why they pay you the big bucks! I didn't change the password, a coworker did (which is why I forgot about it), but yes, it's been changed. Is there a way to specify the new password to use? I scanned through all of the options that I can put into the aws.conf file (as documented here: http://www.cloudera.com/content/cloudera/en/documentation/cloudera-director/latest/PDF/cloudera-dire... but I didn't see anything. Or do I have to change it back to admin:admin, run the update, and change it back?

[#4 · Accepted solution]

For now unfortunately you will to change it back to admin/admin in order to be able to run the update. I will file a feature request on you behalf to make sure we improve this in the next release and document it properly. 

One last thing. My recommendation would be to use Director 1.0.1 that contains some fixes for issues we've discovered after making the release public. 

See: http://cloudera.com/content/cloudera/en/products-and-services/director.html for instructions on how to install Director from packages.

[#5]

Awesome. I reset the password to 'admin' and the update is now working. Thanks!

[#6]

That's great! Can you add a few more details on how you are planning to use Director? It would be useful to make sure future release are aligned with your requirements. 

[#7]

We were planning on using director to build new CDH clusters, and expand existing ones. That's, well, that's pretty much it 🙂