Cloudbreak downscaling fails

[#1]

I am trying to downscale a cluster by removing nodes from a host group containing compute only nodes (no datanode) using a time based alert. However the downscaling fails repeatedly while the upscale operation works without any problems:

Looking at the Cloudbreak logs, it seems like Cloudbreak is worried about the replication factor - which it should not be because I'm only touching compute only nodes. Does anyone know what might be going on in here?

016-04-18 13:42:45,901 [http-nio-8080-exec-4] handleResponseError:581 WARN  o.s.w.c.RestTemplate - [owner:600dbb7a-f175-4684-b16a-fb150997016e] [type:springLog] [id:] [name:] POST request for "http://backend:8089/check_token" resulted in 400 (Bad Request); invoking error handler

2016-04-18 13:59:40,993 [http-nio-8080-exec-1] updateCluster:175 INFO  c.s.c.c.ClusterController - [owner:600dbb7a-f175-4684-b16a-fb150997016e] [type:STACK] [id:56] [name:cluster2] Cluster host adjustment request received. Stack id: 56 

2016-04-18 14:01:06,311 [http-nio-8080-exec-1] verifyNodeCount:523 INFO  c.s.c.s.c.AmbariClusterService - [owner:600dbb7a-f175-4684-b16a-fb150997016e] [type:STACK] [id:56] [name:cluster2] Cannot downscale: replication: 3, adjustment: -1, filtered host size: 3

2016-04-18 14:01:06,312 [http-nio-8080-exec-1] badRequest:42 ERROR c.s.c.c.ExceptionControllerAdvice - [owner:cloudbreak] [type:cloudbreakLog] [id:undefined] [name:cb] There is not enough node to downscale. Check the replication factor and the ApplicationMaster occupation.

com.sequenceiq.cloudbreak.controller.BadRequestException: There is not enough node to downscale. Check the replication factor and the ApplicationMaster occupation.

 at com.sequenceiq.cloudbreak.service.cluster.AmbariClusterService.verifyNodeCount(AmbariClusterService.java:524) ~[cloudbreak.jar!/:na]

 at com.sequenceiq.cloudbreak.service.cluster.AmbariClusterService.collectDownscaleCandidates(AmbariClusterService.java:445) ~[cloudbreak.jar!/:na]

 at com.sequenceiq.cloudbreak.service.cluster.AmbariClusterService.updateHosts(AmbariClusterService.java:218) ~[cloudbreak.jar!/:na]

 at com.sequenceiq.cloudbreak.controller.ClusterController.updateCluster(ClusterController.java:176) ~[cloudbreak.jar!/:na]

 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_79]

 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_79]

 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]

 at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]

 at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215) ~[spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) ~[spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961) [spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:874) [spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at javax.servlet.http.HttpServlet.service(HttpServlet.java:649) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837) [spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilter(WebRequestTraceFilter.java:115) [spring-boot-actuator-1.0.2.RELEASE.jar!/:1.0.2.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextFilterConfiguration$1.doFilterInternal(EndpointWebMvcAutoConfiguration.java:137) [spring-boot-actuator-1.0.2.RELEASE.jar!/:1.0.2.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at com.sequenceiq.cloudbreak.conf.SecurityConfig$ScimAccountGroupReaderFilter.doFilterInternal(SecurityConfig.java:161) [cloudbreak.jar!/:na]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:140) [spring-security-oauth2-2.0.3.RELEASE.jar!/:na]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:85) [spring-boot-actuator-1.0.2.RELEASE.jar!/:1.0.2.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1721) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]

 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]

 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

[#2 · Accepted solution]

There is a bug regarding this that CB does not consider that it's a host group that doesn't contain a datanode. If should be fixed in the following releases.

[#3]

Okay, Thanks!