Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudbreak downscaling fails

Solved Go to solution

Cloudbreak downscaling fails

New Contributor

I am trying to downscale a cluster by removing nodes from a host group containing compute only nodes (no datanode) using a time based alert. However the downscaling fails repeatedly while the upscale operation works without any problems:

3506-screen-shot-2016-04-18-at-110337-am.png

Looking at the Cloudbreak logs, it seems like Cloudbreak is worried about the replication factor - which it should not be because I'm only touching compute only nodes. Does anyone know what might be going on in here?

016-04-18 13:42:45,901 [http-nio-8080-exec-4] handleResponseError:581 WARN  o.s.w.c.RestTemplate - [owner:600dbb7a-f175-4684-b16a-fb150997016e] [type:springLog] [id:] [name:] POST request for "http://backend:8089/check_token" resulted in 400 (Bad Request); invoking error handler

2016-04-18 13:59:40,993 [http-nio-8080-exec-1] updateCluster:175 INFO  c.s.c.c.ClusterController - [owner:600dbb7a-f175-4684-b16a-fb150997016e] [type:STACK] [id:56] [name:cluster2] Cluster host adjustment request received. Stack id: 56 

2016-04-18 14:01:06,311 [http-nio-8080-exec-1] verifyNodeCount:523 INFO  c.s.c.s.c.AmbariClusterService - [owner:600dbb7a-f175-4684-b16a-fb150997016e] [type:STACK] [id:56] [name:cluster2] Cannot downscale: replication: 3, adjustment: -1, filtered host size: 3

2016-04-18 14:01:06,312 [http-nio-8080-exec-1] badRequest:42 ERROR c.s.c.c.ExceptionControllerAdvice - [owner:cloudbreak] [type:cloudbreakLog] [id:undefined] [name:cb] There is not enough node to downscale. Check the replication factor and the ApplicationMaster occupation.

com.sequenceiq.cloudbreak.controller.BadRequestException: There is not enough node to downscale. Check the replication factor and the ApplicationMaster occupation.

 at com.sequenceiq.cloudbreak.service.cluster.AmbariClusterService.verifyNodeCount(AmbariClusterService.java:524) ~[cloudbreak.jar!/:na]

 at com.sequenceiq.cloudbreak.service.cluster.AmbariClusterService.collectDownscaleCandidates(AmbariClusterService.java:445) ~[cloudbreak.jar!/:na]

 at com.sequenceiq.cloudbreak.service.cluster.AmbariClusterService.updateHosts(AmbariClusterService.java:218) ~[cloudbreak.jar!/:na]

 at com.sequenceiq.cloudbreak.controller.ClusterController.updateCluster(ClusterController.java:176) ~[cloudbreak.jar!/:na]

 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_79]

 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_79]

 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_79]

 at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_79]

 at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215) ~[spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) ~[spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870) ~[spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961) [spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:874) [spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at javax.servlet.http.HttpServlet.service(HttpServlet.java:649) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837) [spring-webmvc-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilter(WebRequestTraceFilter.java:115) [spring-boot-actuator-1.0.2.RELEASE.jar!/:1.0.2.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextFilterConfiguration$1.doFilterInternal(EndpointWebMvcAutoConfiguration.java:137) [spring-boot-actuator-1.0.2.RELEASE.jar!/:1.0.2.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at com.sequenceiq.cloudbreak.conf.SecurityConfig$ScimAccountGroupReaderFilter.doFilterInternal(SecurityConfig.java:161) [cloudbreak.jar!/:na]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:140) [spring-security-oauth2-2.0.3.RELEASE.jar!/:na]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) [spring-security-web-3.2.3.RELEASE.jar!/:3.2.3.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:85) [spring-boot-actuator-1.0.2.RELEASE.jar!/:1.0.2.RELEASE]

 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108) [spring-web-4.0.3.RELEASE.jar!/:4.0.3.RELEASE]

 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1721) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679) [tomcat-embed-core-7.0.52.jar!/:7.0.52]

 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]

 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]

 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]


1 ACCEPTED SOLUTION

Accepted Solutions

Re: Cloudbreak downscaling fails

Expert Contributor

There is a bug regarding this that CB does not consider that it's a host group that doesn't contain a datanode. If should be fixed in the following releases.

2 REPLIES 2

Re: Cloudbreak downscaling fails

Expert Contributor

There is a bug regarding this that CB does not consider that it's a host group that doesn't contain a datanode. If should be fixed in the following releases.

Re: Cloudbreak downscaling fails

New Contributor

Okay, Thanks!

Don't have an account?
Coming from Hortonworks? Activate your account here