Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: Could not upscale: cbsandbox1

Explorer

Hi,

Cloudbreak is failing to upscale the cluster with the below error, we could see that the VM's are spin up and running in Azure portal under the resource group. Tried to find any leads on the error but couldn't find any, Any help would be appreciated.

Error:

/cbreak_cloudbreak_1 | 2018-02-13 07:22:48,811 [RxIoScheduler-12] log:55 INFO c.m.a.m.r.Deployments createOrUpdate (poll) - [owner:spring] [type:springLog] [id:] [name:] --> GET https://management.azure.com/subscriptions/<subscription_id>/resourcegroups/cbsandbox1/providers/Mic... /cbreak_cloudbreak_1 | 2018-02-13 07:22:48,946 [RxIoScheduler-12] log:55 INFO c.m.a.m.r.Deployments createOrUpdate (poll) - [owner:spring] [type:springLog] [id:] [name:] <-- 200 OK https://management.azure.com/subscriptions/<subscription_id>/resourcegroups/cbsandbox1/providers/Mic... (135 ms, 389-byte body) /cbreak_cloudbreak_1 | 2018-02-13 07:22:48,960 [reactorDispatcher-40] accept:140 DEBUG c.s.c.c.f.Flow2Handler - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:springLog] [id:1] [name:cbllapdev30] flow control event arrived: key: UPSCALESTACKRESULT_ERROR, flowid: 08bd0afd-627c-42d4-9e36-4d8232961d61, payload: CloudPlatformResult{status=FAILED, statusReason='Could not upscale: cbsandbox1 ', errorDetails=com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: Could not upscale: cbsandbox1 , request=UpscaleStackRequest{resourceList=[CloudResource{type=ARM_TEMPLATE, status=CREATED, name='cbsandbox1', reference='null', group='null', persistent='true'}]}} /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,110 [reactorDispatcher-40] execute:73 INFO c.s.c.c.f.AbstractAction - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] Stack: 1, flow state: ADD_INSTANCES_STATE, phase: doExec, execution time 190 sec /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,111 [reactorDispatcher-40] handleStackUpscaleFailure:201 ERROR c.s.c.c.f.s.u.StackUpscaleService - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] Exception during the upscale of stack /cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: Could not upscale: cbsandbox1 /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.azure.AzureResourceConnector.upscale(AzureResourceConnector.java:220) /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.UpscaleStackHandler.accept(UpscaleStackHandler.java:64) /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.UpscaleStackHandler.accept(UpscaleStackHandler.java:31) /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.UpscaleStackHandler$FastClassBySpringCGLIB$4d3f5688.invoke(<generated>) /cbreak_cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) /cbreak_cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.UpscaleStackHandler$EnhancerBySpringCGLIB$78e983cb.accept(<generated>) /cbreak_cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317) /cbreak_cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310) /cbreak_cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72) /cbreak_cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51) /cbreak_cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591) /cbreak_cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63) /cbreak_cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160) /cbreak_cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74) /cbreak_cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) /cbreak_cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) /cbreak_cloudbreak_1 | at java.lang.Thread.run(Thread.java:745) /cbreak_cloudbreak_1 | Caused by: com.microsoft.azure.CloudException: Async operation failed with provisioning state: Failed /cbreak_cloudbreak_1 | at com.microsoft.azure.AzureClient$1$2.call(AzureClient.java:187) /cbreak_cloudbreak_1 | at com.microsoft.azure.AzureClient$1$2.call(AzureClient.java:176) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69) /cbreak_cloudbreak_1 | at rx.internal.producers.SingleProducer.request(SingleProducer.java:65) /cbreak_cloudbreak_1 | at rx.Subscriber.setProducer(Subscriber.java:211) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorSingle$ParentSubscriber.onCompleted(OperatorSingle.java:110) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorTake$1.onNext(OperatorTake.java:80) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeFilter$FilterSubscriber.onNext(OnSubscribeFilter.java:76) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:395) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:355) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:846) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) /cbreak_cloudbreak_1 | at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$RequestArbiter.request(RxJavaCallAdapterFactory.java:173) /cbreak_cloudbreak_1 | at rx.Subscriber.setProducer(Subscriber.java:211) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) /cbreak_cloudbreak_1 | at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:152) /cbreak_cloudbreak_1 | at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:138) /cbreak_cloudbreak_1 | at rx.Observable.unsafeSubscribe(Observable.java:10142) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeRedo$2$1.onNext(OnSubscribeRedo.java:244) /cbreak_cloudbreak_1 | at rx.internal.util.ScalarSynchronousObservable$ScalarAsyncProducer.call(ScalarSynchronousObservable.java:200) /cbreak_cloudbreak_1 | at rx.internal.util.ScalarSynchronousObservable$2$1.call(ScalarSynchronousObservable.java:114) /cbreak_cloudbreak_1 | at rx.internal.schedulers.CachedThreadScheduler$EventLoopWorker$1.call(CachedThreadScheduler.java:230) /cbreak_cloudbreak_1 | at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) /cbreak_cloudbreak_1 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) /cbreak_cloudbreak_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266) /cbreak_cloudbreak_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) /cbreak_cloudbreak_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) /cbreak_cloudbreak_1 | ... 3 common frames omitted /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,262 [reactorDispatcher-40] fireEventAndLog:25 DEBUG c.s.c.c.f.s.FlowMessageService - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] STACK_INFRASTRUCTURE_UPDATE_FAILED [STACK_FLOW_STEP]. /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,263 [reactorDispatcher-40] fireCloudbreakEvent:52 INFO c.s.c.s.e.DefaultCloudbreakEventService - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] Firing Cloudbreak event: CloudbreakEventData{entityId=1, eventType='AVAILABLE', eventMessage='Stack update failed. Reason: Could not upscale: cbsandbox1 '} /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,263 [reactorDispatcher-40] sendEvent:112 INFO c.s.c.c.f.AbstractAction - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] Triggering event: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@66325a31 /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,264 [reactorDispatcher-44] accept:33 INFO c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Handling cloudbreak event: Event{id=null, headers=null, replyTo=null, key=CLOUDBREAK_EVENT, data=CloudbreakEventData{entityId=1, eventType='AVAILABLE', eventMessage='Stack update failed. Reason: Could not upscale: cbsandbox1 '}} /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,264 [reactorDispatcher-40] stateChanged:134 DEBUG c.s.c.c.f.c.AbstractFlowConfiguration - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] state changed from ObjectState [getIds()=[ADD_INSTANCES_STATE], getClass()=class org.springframework.statemachine.state.ObjectState, hashCode()=57375442, toString()=AbstractState [id=ADD_INSTANCES_STATE, pseudoState=null, deferred=null, entryActions=[com.sequenceiq.cloudbreak.core.flow2.stack.upscale.StackUpscaleActions$1@2ceee4b6], exitActions=null, regions=[], submachine=null]] to ObjectState [getIds()=[UPSCALE_FAILED_STATE], getClass()=class org.springframework.statemachine.state.ObjectState, hashCode()=379254949, toString()=AbstractState [id=UPSCALE_FAILED_STATE, pseudoState=null, deferred=null, entryActions=[com.sequenceiq.cloudbreak.core.flow2.stack.upscale.StackUpscaleActions$9@13c8ac77], exitActions=null, regions=[], submachine=null]] /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,264 [reactorDispatcher-42] accept:140 DEBUG c.s.c.c.f.Flow2Handler - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:7] [name:cbtest1] flow control event arrived: key: UPSCALEFAILHANDLED, flowid: 08bd0afd-627c-42d4-9e36-4d8232961d61, payload: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@66325a31

1 REPLY 1

Expert Contributor

Hi @Cibi Chakaravarthi,

Could you please attach the log from the beginning of the scaling? And please attach the log and don't paste it because it's hard to read this way.