Cannot grow cluster (javax.ws.rs.NotAuthorizedException)

[#1]

Verified that no changes were made to the director password, related AWS keys, and I have not hit my limit of instances.

[2015-04-08 15:44:46] WARN [qtp949320138-46] - o.e.jetty.servlet.ServletHandler: /api/v1/environments/development/deployments/development/clusters/development-cluster
javax.ws.rs.NotAuthorizedException: null

Version: Cloudera Director 1.0.2

[#2]

I took a snapshot of the server before upgrading to the lastest cloudera director hoping that would help, but it has not. In fact when I modify the cluster, add worker nodes, and click the 'Continue' button there is no request to the API to initiate the call to create new instances (at least not that I can see in the web inspector). There's also no change in the application.log that would suggest an error occured.

This is pretty dissapointing.

Version: Cloudera Director 1.1.1

[#3]

Unfortunately this is a known upgrade bug that we will be fixing in the 1.1.2 patch release that will be available soon. The issue has to do with the fact that the list of parcel repositories is empty on the cluster created via 1.0.x. 

You should be able to grow a cluster created with 1.1.1 in the first place as expected. Is it possible for you to migrate the data from the existing cluster? 

[#4]

I can revert back to 1.0.2 via the snapshot I created if you can explain how to resolve the authorization errors.

I really dont have the time to create and bootstrap (with our code) a new cluster, nor do I want to spend the additional money of having two clusters up and running for a several hours to do that.

How soon will the 1.1.2 patch be released?

[#5]

Reverting to 1.0.2 may work. That authorization exception should not happen if you can successfuly login. Can you give it another try in incognito mode? (just to make sure there are no other cookies in your browser) 

[#6]

I could definitely login without any issue. I will try reverting to the snapshot and using incognito mode.

[#7]

So I reverted back to the older version that was used to create the cluster, logged in with incognito, and I get the same error. Tried two other browsers I've never used with the cloudera director, or any of the cloudera clusters for that matter, and I get the same error in all cases.

What are my options going forward? I will gladly capture any logs you need.

[#8]

Hi,

I'd like to get a little more context about situation.

How was your cluster created?

What commands are you submitting to Director? How are you submitting them?

Do all commands fail w/ the authorization exception or just when you try to grow the cluster?

Can you give a longer log? I'd like to see what's happening immediately preceding the exception.

Thanks!

David

[#9]

The cluster was created with cloudera director, in fact I have three clusters created with director. The only commands I have attempted to submit are to grow the number of workers in my development cluster. This is simply done by going into director web UI, choosing `grow cluster` from the actions dropdown, changing the number of workers from 2 to any number higher than 2 and clicking `continue`. A modal popup asks `This will start up 2 EC2 instance(s). Are you sure you want to proceed?`, I click `OK` and immediately Im shown a second modal that says `Sorry, an error has occured. Please try again later. Status: 500, Reason: Unknown`

Version: Cloudera Director 1.0.2

Here are all the logs generated when I perform those actions. I've started from a completely logged out state, logged in with the admin user and performed the actions exactly as I've described above. (this is /var/log/cloudera-director-server/application.log)

[2015-04-08 20:55:16] INFO  [qtp1002075522-45] - c.c.l.a.r.v.AuthenticationResourceV1: Logging in admin via API
[2015-04-08 20:55:16] INFO  [qtp1002075522-45] - o.s.b.a.audit.listener.AuditListener: AuditEvent [timestamp=Wed Apr 08 20:55:16 UTC 2015, principal=admin, type=AUTHENTICATION_SUCCESS, data={}]
[2015-04-08 20:55:32] INFO  [qtp1002075522-44] - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating parcels and version compatibility
[2015-04-08 20:55:33] WARN  [qtp1002075522-44] - o.e.jetty.servlet.ServletHandler:
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is javax.ws.rs.NotAuthorizedException
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:973) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:874) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:758) ~[javax.servlet-3.0.0.v201112011016.jar!/:na]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) ~[javax.servlet-3.0.0.v201112011016.jar!/:na]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) ~[spring-boot-actuator-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:280) ~[spring-boot-actuator-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:89) ~[spring-boot-actuator-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) [jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) [jetty-security-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) [jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.Server.handle(Server.java:370) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) [jetty-http-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) [jetty-http-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) [jetty-io-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) [jetty-io-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Caused by: javax.ws.rs.NotAuthorizedException: null
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_65]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_65]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_65]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_65]
	at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:452) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.checkResponse(ClientProxyImpl.java:298) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:586) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:552) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at com.sun.proxy.$Proxy149.getConfig(Unknown Source) ~[na:na]
	at com.cloudera.launchpad.bootstrap.validators.GenericClusterTemplateValidator.validate(GenericClusterTemplateValidator.java:82) ~[launchpad-bootstrap-1.0.2.jar!/:1.0.2]
	at com.cloudera.launchpad.model.ValidatorRegistry.validate(ValidatorRegistry.java:82) ~[launchpad-model-1.0.2.jar!/:1.0.2]
	at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1.validate(ClustersResourceV1.java:578) ~[launchpad-api-resources-1.0.2.jar!/:1.0.2]
	at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1.update(ClustersResourceV1.java:304) ~[launchpad-api-resources-1.0.2.jar!/:1.0.2]
	at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1$$FastClassBySpringCGLIB$$81e120c8.invoke(<generated>) ~[spring-core-4.0.6.RELEASE.jar!/:1.0.2]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708) ~[spring-aop-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) ~[spring-security-core-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644) ~[spring-aop-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1$$EnhancerBySpringCGLIB$$40df15a2.update(<generated>) ~[spring-core-4.0.6.RELEASE.jar!/:1.0.2]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_65]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_65]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_65]
	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_65]
	at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	... 69 common frames omitted
[2015-04-08 20:55:33] WARN  [qtp1002075522-44] - o.e.jetty.servlet.ServletHandler: /api/v1/environments/development/deployments/development/clusters/development-cluster
javax.ws.rs.NotAuthorizedException: null
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_65]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_65]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_65]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_65]
	at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:452) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.checkResponse(ClientProxyImpl.java:298) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:586) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:552) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
	at com.sun.proxy.$Proxy149.getConfig(Unknown Source) ~[na:na]
	at com.cloudera.launchpad.bootstrap.validators.GenericClusterTemplateValidator.validate(GenericClusterTemplateValidator.java:82) ~[launchpad-bootstrap-1.0.2.jar!/:1.0.2]
	at com.cloudera.launchpad.model.ValidatorRegistry.validate(ValidatorRegistry.java:82) ~[launchpad-model-1.0.2.jar!/:1.0.2]
	at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1.validate(ClustersResourceV1.java:578) ~[launchpad-api-resources-1.0.2.jar!/:1.0.2]
	at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1.update(ClustersResourceV1.java:304) ~[launchpad-api-resources-1.0.2.jar!/:1.0.2]
	at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1$$FastClassBySpringCGLIB$$81e120c8.invoke(<generated>) ~[spring-core-4.0.6.RELEASE.jar!/:1.0.2]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708) ~[spring-aop-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) ~[spring-security-core-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644) ~[spring-aop-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at com.cloudera.launchpad.api.resources.v1.ClustersResourceV1$$EnhancerBySpringCGLIB$$40df15a2.update(<generated>) ~[spring-core-4.0.6.RELEASE.jar!/:1.0.2]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_65]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_65]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_65]
	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_65]
	at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:874) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:758) ~[javax.servlet-3.0.0.v201112011016.jar!/:na]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837) ~[spring-webmvc-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) ~[javax.servlet-3.0.0.v201112011016.jar!/:na]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) ~[spring-boot-actuator-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:280) ~[spring-boot-actuator-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) ~[spring-security-web-3.2.4.RELEASE.jar!/:3.2.4.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:89) ~[spring-boot-actuator-1.1.5.RELEASE.jar!/:1.1.5.RELEASE]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.0.6.RELEASE.jar!/:4.0.6.RELEASE]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) ~[jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) [jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) [jetty-security-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) [jetty-servlet-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.Server.handle(Server.java:370) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) [jetty-http-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) [jetty-http-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) [jetty-io-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) [jetty-io-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-8.1.15.v20140411.jar!/:8.1.15.v20140411]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]

Just to be clear I think director is a fantastic product when it works, I've preached its proficiency at several meetups for people looking to create hadoop clusters on AWS even against other solutions like Chef (which I happen to also be a huge advocate for). However, when it fails in bizarre and unexpected ways due to no changes in configuration on the users part it begins to feel fragile and Im left wondering if Ive invested my time into the wrong solution.

[#10 · Accepted solution]

I'm happy to hear that you've had good success with Cloudera Director. We appreciate you taking the time to try it out. We take all feedback to heart and strive to improve the product.

Have you changed the user/password on your Cloudera Manager instance for this cluster? It looks like the failure occurs during cluster template validation. During this time, Cloudera Director calls out to Cloudera Manager to validate that the product versions match.