Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudbreak 2.9.0 fails to deploy on openstack

Highlighted

Cloudbreak 2.9.0 fails to deploy on openstack

New Contributor

Hi,

I'm trying to deploy an HDP cluster on openstack using Cloudbreak 2.9.0. However the deployment fails pretty fast and I see the following entries in the logs of cloudbreak:


cloudbreak_1 | 2019-05-20 13:25:34,720 [reactorDispatcher-52] handleImageNotFound:54 ERROR c.s.c.c.o.c.OpenStackImageVerifier - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACK] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] OpenStack image: cb-hdp-31-1901151759 not found
cloudbreak_1 | 2019-05-20 13:25:34,816 [reactorDispatcher-52] handleImageNotFound:59 WARN c.s.c.c.o.c.OpenStackImageVerifier - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACK] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Exception occured during listing openstack images on V2 API. Falling back to V1 API.
cloudbreak_1 | javax.ws.rs.ProcessingException: Error reading entity from input stream.
cloudbreak_1 | at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:889)
cloudbreak_1 | at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:808)
cloudbreak_1 | at org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:321)
cloudbreak_1 | at org.glassfish.jersey.client.InboundJaxrsResponse$1.call(InboundJaxrsResponse.java:115)
cloudbreak_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
cloudbreak_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
cloudbreak_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:229)
cloudbreak_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:389)
cloudbreak_1 | at org.glassfish.jersey.client.InboundJaxrsResponse.runInScopeIfPossible(InboundJaxrsResponse.java:264)
cloudbreak_1 | at org.glassfish.jersey.client.InboundJaxrsResponse.readEntity(InboundJaxrsResponse.java:112)
cloudbreak_1 | at org.openstack4j.connectors.jersey2.HttpResponseImpl.readEntity(HttpResponseImpl.java:113)
cloudbreak_1 | at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:66)
cloudbreak_1 | at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:24)
cloudbreak_1 | at org.openstack4j.connectors.jersey2.HttpResponseImpl.getEntity(HttpResponseImpl.java:63)
cloudbreak_1 | at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:225)
cloudbreak_1 | at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:207)
cloudbreak_1 | at org.openstack4j.openstack.image.v2.internal.ImageServiceImpl.list(ImageServiceImpl.java:45)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.handleImageNotFound(OpenStackImageVerifier.java:57)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.getStatusFromImages(OpenStackImageVerifier.java:44)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.getStatus(OpenStackImageVerifier.java:33)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.exist(OpenStackImageVerifier.java:27)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackSetup.prepareImage(OpenStackSetup.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler.accept(PrepareImageHandler.java:47)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler.accept(PrepareImageHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler$$FastClassBySpringCGLIB$$aaf961cd.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler$$EnhancerBySpringCGLIB$$143904db.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
cloudbreak_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
cloudbreak_1 | at java.base/java.lang.Thread.run(Thread.java:844)
cloudbreak_1 | Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
cloudbreak_1 | at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 644] (through reference chain: org.openstack4j.openstack.image.v2.domain.GlanceImage$Images["images"]->java.util.ArrayList[0]->org.openstack4j.openstack.image.v2.domain.GlanceImage["locations"]->java.util.ArrayList[0])
cloudbreak_1 | at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
cloudbreak_1 | at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)
cloudbreak_1 | at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)
cloudbreak_1 | at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.StdDeserializer._parseString(StdDeserializer.java:569)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:203)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:169)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:21)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:136)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:286)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:27)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:136)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
cloudbreak_1 | at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:1574)
cloudbreak_1 | at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:965)
cloudbreak_1 | at org.glassfish.jersey.jackson.internal.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:838)
cloudbreak_1 | at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:257)
cloudbreak_1 | at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:236)
cloudbreak_1 | at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:156)
cloudbreak_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1091)
cloudbreak_1 | at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:874)
cloudbreak_1 | ... 44 common frames omitted
cloudbreak_1 | 2019-05-20 13:25:34,925 [reactorDispatcher-53] buildLogContextForReactorHandler:69 INFO c.s.c.l.LogContextAspects - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACK] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] A Reactor event handler's 'accept' method has been intercepted: execution(Flow2Handler.accept(..)), MDC logger context is built.
cloudbreak_1 | 2019-05-20 13:25:34,931 [reactorDispatcher-53] proceed:132 INFO c.s.c.a.PermissionCheckingUtils - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACK] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Return value is null, method signature: public abstract void com.sequenceiq.cloudbreak.repository.FlowLogRepository.updateLastLogStatusInFlow(java.lang.Long,com.sequenceiq.cloudbreak.domain.StateStatus)
cloudbreak_1 | 2019-05-20 13:25:34,951 [reactorDispatcher-53] execute:87 INFO c.s.c.c.f.AbstractAction - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACKVIEW] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Stack: 1, flow state: IMAGESETUP_STATE, phase: doExec, execution time 1 sec
cloudbreak_1 | 2019-05-20 13:25:34,953 [reactorDispatcher-53] handleStackCreationFailure:208 ERROR c.s.c.c.f.s.p.a.StackCreationService - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACKVIEW] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Error during stack creation flow:
cloudbreak_1 | org.openstack4j.api.exceptions.ServerResponseException: Internal Server Error
cloudbreak_1 | at org.openstack4j.core.transport.HttpExceptionHandler.mapException(HttpExceptionHandler.java:40)
cloudbreak_1 | at org.openstack4j.core.transport.HttpExceptionHandler.mapException(HttpExceptionHandler.java:23)
cloudbreak_1 | at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:51)
cloudbreak_1 | at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:24)
cloudbreak_1 | at org.openstack4j.connectors.jersey2.HttpResponseImpl.getEntity(HttpResponseImpl.java:63)
cloudbreak_1 | at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:225)
cloudbreak_1 | at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:207)
cloudbreak_1 | at org.openstack4j.openstack.image.internal.ImageServiceImpl.list(ImageServiceImpl.java:62)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.handleImageNotFound(OpenStackImageVerifier.java:60)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.getStatusFromImages(OpenStackImageVerifier.java:44)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.getStatus(OpenStackImageVerifier.java:33)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.exist(OpenStackImageVerifier.java:27)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackSetup.prepareImage(OpenStackSetup.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler.accept(PrepareImageHandler.java:47)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler.accept(PrepareImageHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler$$FastClassBySpringCGLIB$$aaf961cd.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler$$EnhancerBySpringCGLIB$$143904db.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
cloudbreak_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
cloudbreak_1 | at java.base/java.lang.Thread.run(Thread.java:844)
cloudbreak_1 | 2019-05-20 13:25:35,123 [reactorDispatcher-53] fireCloudbreakEvent:51 INFO c.s.c.s.e.DefaultCloudbreakEventService - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACKVIEW] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Firing Cloudbreak event: entityId: 1, type: CREATE_FAILED, message: Infrastructure creation failed. Reason: Internal Server Error



Any help is appreciated.


Cheers,

Martin


4 REPLIES 4

Re: Cloudbreak 2.9.0 fails to deploy on openstack

Expert Contributor

Hi @Martin Prodanov,

Looks like the image cannot be automatically imported into your openstack. You can download it and manually import from https://public-repo-1.hortonworks.com/HDP/cloudbreak/cb-hdp-31-1901151759.img

Re: Cloudbreak 2.9.0 fails to deploy on openstack

New Contributor

Hi,

thanks for the hint @mmolnar. I tried but it did not help. I still see the same exception. Do you have to specify some particular name in OS for the image?


Cheers,

Martin

Re: Cloudbreak 2.9.0 fails to deploy on openstack

Expert Contributor

Hi @Martin Prodanov,

it should be called "cb-hdp-31-1901151759". CB looks for images by name on openstack.

Re: Cloudbreak 2.9.0 fails to deploy on openstack

New Contributor

I tried but if fails with the same exception.