Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: Could not upscale: cbsandbox1

Highlighted

com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: Could not upscale: cbsandbox1

New Contributor

Hi,

Cloudbreak is failing to upscale the cluster with the below error, we could see that the VM's are spin up and running in Azure portal under the resource group. Tried to find any leads on the error but couldn't find any, Any help would be appreciated.

Error:

/cbreak_cloudbreak_1 | 2018-02-13 07:22:48,811 [RxIoScheduler-12] log:55 INFO c.m.a.m.r.Deployments createOrUpdate (poll) - [owner:spring] [type:springLog] [id:] [name:] --> GET https://management.azure.com/subscriptions/<subscription_id>/resourcegroups/cbsandbox1/providers/Mic... /cbreak_cloudbreak_1 | 2018-02-13 07:22:48,946 [RxIoScheduler-12] log:55 INFO c.m.a.m.r.Deployments createOrUpdate (poll) - [owner:spring] [type:springLog] [id:] [name:] <-- 200 OK https://management.azure.com/subscriptions/<subscription_id>/resourcegroups/cbsandbox1/providers/Mic... (135 ms, 389-byte body) /cbreak_cloudbreak_1 | 2018-02-13 07:22:48,960 [reactorDispatcher-40] accept:140 DEBUG c.s.c.c.f.Flow2Handler - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:springLog] [id:1] [name:cbllapdev30] flow control event arrived: key: UPSCALESTACKRESULT_ERROR, flowid: 08bd0afd-627c-42d4-9e36-4d8232961d61, payload: CloudPlatformResult{status=FAILED, statusReason='Could not upscale: cbsandbox1 ', errorDetails=com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: Could not upscale: cbsandbox1 , request=UpscaleStackRequest{resourceList=[CloudResource{type=ARM_TEMPLATE, status=CREATED, name='cbsandbox1', reference='null', group='null', persistent='true'}]}} /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,110 [reactorDispatcher-40] execute:73 INFO c.s.c.c.f.AbstractAction - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] Stack: 1, flow state: ADD_INSTANCES_STATE, phase: doExec, execution time 190 sec /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,111 [reactorDispatcher-40] handleStackUpscaleFailure:201 ERROR c.s.c.c.f.s.u.StackUpscaleService - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] Exception during the upscale of stack /cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: Could not upscale: cbsandbox1 /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.azure.AzureResourceConnector.upscale(AzureResourceConnector.java:220) /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.UpscaleStackHandler.accept(UpscaleStackHandler.java:64) /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.UpscaleStackHandler.accept(UpscaleStackHandler.java:31) /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.UpscaleStackHandler$FastClassBySpringCGLIB$4d3f5688.invoke(<generated>) /cbreak_cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) /cbreak_cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) /cbreak_cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) /cbreak_cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.UpscaleStackHandler$EnhancerBySpringCGLIB$78e983cb.accept(<generated>) /cbreak_cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317) /cbreak_cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310) /cbreak_cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72) /cbreak_cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51) /cbreak_cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591) /cbreak_cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63) /cbreak_cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160) /cbreak_cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74) /cbreak_cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) /cbreak_cloudbreak_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) /cbreak_cloudbreak_1 | at java.lang.Thread.run(Thread.java:745) /cbreak_cloudbreak_1 | Caused by: com.microsoft.azure.CloudException: Async operation failed with provisioning state: Failed /cbreak_cloudbreak_1 | at com.microsoft.azure.AzureClient$1$2.call(AzureClient.java:187) /cbreak_cloudbreak_1 | at com.microsoft.azure.AzureClient$1$2.call(AzureClient.java:176) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69) /cbreak_cloudbreak_1 | at rx.internal.producers.SingleProducer.request(SingleProducer.java:65) /cbreak_cloudbreak_1 | at rx.Subscriber.setProducer(Subscriber.java:211) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorSingle$ParentSubscriber.onCompleted(OperatorSingle.java:110) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorTake$1.onNext(OperatorTake.java:80) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeFilter$FilterSubscriber.onNext(OnSubscribeFilter.java:76) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:395) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:355) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:846) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:511) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:466) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:244) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) /cbreak_cloudbreak_1 | at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$RequestArbiter.request(RxJavaCallAdapterFactory.java:173) /cbreak_cloudbreak_1 | at rx.Subscriber.setProducer(Subscriber.java:211) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102) /cbreak_cloudbreak_1 | at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:152) /cbreak_cloudbreak_1 | at retrofit2.adapter.rxjava.RxJavaCallAdapterFactory$CallOnSubscribe.call(RxJavaCallAdapterFactory.java:138) /cbreak_cloudbreak_1 | at rx.Observable.unsafeSubscribe(Observable.java:10142) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248) /cbreak_cloudbreak_1 | at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77) /cbreak_cloudbreak_1 | at rx.internal.operators.OnSubscribeRedo$2$1.onNext(OnSubscribeRedo.java:244) /cbreak_cloudbreak_1 | at rx.internal.util.ScalarSynchronousObservable$ScalarAsyncProducer.call(ScalarSynchronousObservable.java:200) /cbreak_cloudbreak_1 | at rx.internal.util.ScalarSynchronousObservable$2$1.call(ScalarSynchronousObservable.java:114) /cbreak_cloudbreak_1 | at rx.internal.schedulers.CachedThreadScheduler$EventLoopWorker$1.call(CachedThreadScheduler.java:230) /cbreak_cloudbreak_1 | at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) /cbreak_cloudbreak_1 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) /cbreak_cloudbreak_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266) /cbreak_cloudbreak_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) /cbreak_cloudbreak_1 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) /cbreak_cloudbreak_1 | ... 3 common frames omitted /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,262 [reactorDispatcher-40] fireEventAndLog:25 DEBUG c.s.c.c.f.s.FlowMessageService - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] STACK_INFRASTRUCTURE_UPDATE_FAILED [STACK_FLOW_STEP]. /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,263 [reactorDispatcher-40] fireCloudbreakEvent:52 INFO c.s.c.s.e.DefaultCloudbreakEventService - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] Firing Cloudbreak event: CloudbreakEventData{entityId=1, eventType='AVAILABLE', eventMessage='Stack update failed. Reason: Could not upscale: cbsandbox1 '} /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,263 [reactorDispatcher-40] sendEvent:112 INFO c.s.c.c.f.AbstractAction - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] Triggering event: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@66325a31 /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,264 [reactorDispatcher-44] accept:33 INFO c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Handling cloudbreak event: Event{id=null, headers=null, replyTo=null, key=CLOUDBREAK_EVENT, data=CloudbreakEventData{entityId=1, eventType='AVAILABLE', eventMessage='Stack update failed. Reason: Could not upscale: cbsandbox1 '}} /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,264 [reactorDispatcher-40] stateChanged:134 DEBUG c.s.c.c.f.c.AbstractFlowConfiguration - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:1] [name:cbllapdev30] state changed from ObjectState [getIds()=[ADD_INSTANCES_STATE], getClass()=class org.springframework.statemachine.state.ObjectState, hashCode()=57375442, toString()=AbstractState [id=ADD_INSTANCES_STATE, pseudoState=null, deferred=null, entryActions=[com.sequenceiq.cloudbreak.core.flow2.stack.upscale.StackUpscaleActions$1@2ceee4b6], exitActions=null, regions=[], submachine=null]] to ObjectState [getIds()=[UPSCALE_FAILED_STATE], getClass()=class org.springframework.statemachine.state.ObjectState, hashCode()=379254949, toString()=AbstractState [id=UPSCALE_FAILED_STATE, pseudoState=null, deferred=null, entryActions=[com.sequenceiq.cloudbreak.core.flow2.stack.upscale.StackUpscaleActions$9@13c8ac77], exitActions=null, regions=[], submachine=null]] /cbreak_cloudbreak_1 | 2018-02-13 07:22:49,264 [reactorDispatcher-42] accept:140 DEBUG c.s.c.c.f.Flow2Handler - [owner:d312e73a-f6dc-4e83-9452-bde66b18791f] [type:STACK] [id:7] [name:cbtest1] flow control event arrived: key: UPSCALEFAILHANDLED, flowid: 08bd0afd-627c-42d4-9e36-4d8232961d61, payload: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@66325a31

1 REPLY 1

Re: com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: Could not upscale: cbsandbox1

Expert Contributor

Hi @Cibi Chakaravarthi,

Could you please attach the log from the beginning of the scaling? And please attach the log and don't paste it because it's hard to read this way.

Don't have an account?
Coming from Hortonworks? Activate your account here