Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Cloudbreak 2.9.0 fails to deploy on openstack

New Contributor

Hi,

I'm trying to deploy an HDP cluster on openstack using Cloudbreak 2.9.0. However the deployment fails pretty fast and I see the following entries in the logs of cloudbreak:


cloudbreak_1 | 2019-05-20 13:25:34,720 [reactorDispatcher-52] handleImageNotFound:54 ERROR c.s.c.c.o.c.OpenStackImageVerifier - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACK] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] OpenStack image: cb-hdp-31-1901151759 not found
cloudbreak_1 | 2019-05-20 13:25:34,816 [reactorDispatcher-52] handleImageNotFound:59 WARN c.s.c.c.o.c.OpenStackImageVerifier - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACK] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Exception occured during listing openstack images on V2 API. Falling back to V1 API.
cloudbreak_1 | javax.ws.rs.ProcessingException: Error reading entity from input stream.
cloudbreak_1 | at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:889)
cloudbreak_1 | at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:808)
cloudbreak_1 | at org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:321)
cloudbreak_1 | at org.glassfish.jersey.client.InboundJaxrsResponse$1.call(InboundJaxrsResponse.java:115)
cloudbreak_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
cloudbreak_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
cloudbreak_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:229)
cloudbreak_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:389)
cloudbreak_1 | at org.glassfish.jersey.client.InboundJaxrsResponse.runInScopeIfPossible(InboundJaxrsResponse.java:264)
cloudbreak_1 | at org.glassfish.jersey.client.InboundJaxrsResponse.readEntity(InboundJaxrsResponse.java:112)
cloudbreak_1 | at org.openstack4j.connectors.jersey2.HttpResponseImpl.readEntity(HttpResponseImpl.java:113)
cloudbreak_1 | at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:66)
cloudbreak_1 | at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:24)
cloudbreak_1 | at org.openstack4j.connectors.jersey2.HttpResponseImpl.getEntity(HttpResponseImpl.java:63)
cloudbreak_1 | at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:225)
cloudbreak_1 | at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:207)
cloudbreak_1 | at org.openstack4j.openstack.image.v2.internal.ImageServiceImpl.list(ImageServiceImpl.java:45)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.handleImageNotFound(OpenStackImageVerifier.java:57)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.getStatusFromImages(OpenStackImageVerifier.java:44)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.getStatus(OpenStackImageVerifier.java:33)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.exist(OpenStackImageVerifier.java:27)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackSetup.prepareImage(OpenStackSetup.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler.accept(PrepareImageHandler.java:47)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler.accept(PrepareImageHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler$$FastClassBySpringCGLIB$$aaf961cd.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler$$EnhancerBySpringCGLIB$$143904db.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
cloudbreak_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
cloudbreak_1 | at java.base/java.lang.Thread.run(Thread.java:844)
cloudbreak_1 | Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
cloudbreak_1 | at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 644] (through reference chain: org.openstack4j.openstack.image.v2.domain.GlanceImage$Images["images"]->java.util.ArrayList[0]->org.openstack4j.openstack.image.v2.domain.GlanceImage["locations"]->java.util.ArrayList[0])
cloudbreak_1 | at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
cloudbreak_1 | at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)
cloudbreak_1 | at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)
cloudbreak_1 | at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.StdDeserializer._parseString(StdDeserializer.java:569)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:203)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:169)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.StringCollectionDeserializer.deserialize(StringCollectionDeserializer.java:21)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:136)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:286)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:27)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:136)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
cloudbreak_1 | at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
cloudbreak_1 | at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:1574)
cloudbreak_1 | at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:965)
cloudbreak_1 | at org.glassfish.jersey.jackson.internal.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:838)
cloudbreak_1 | at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:257)
cloudbreak_1 | at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:236)
cloudbreak_1 | at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:156)
cloudbreak_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1091)
cloudbreak_1 | at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:874)
cloudbreak_1 | ... 44 common frames omitted
cloudbreak_1 | 2019-05-20 13:25:34,925 [reactorDispatcher-53] buildLogContextForReactorHandler:69 INFO c.s.c.l.LogContextAspects - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACK] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] A Reactor event handler's 'accept' method has been intercepted: execution(Flow2Handler.accept(..)), MDC logger context is built.
cloudbreak_1 | 2019-05-20 13:25:34,931 [reactorDispatcher-53] proceed:132 INFO c.s.c.a.PermissionCheckingUtils - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACK] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Return value is null, method signature: public abstract void com.sequenceiq.cloudbreak.repository.FlowLogRepository.updateLastLogStatusInFlow(java.lang.Long,com.sequenceiq.cloudbreak.domain.StateStatus)
cloudbreak_1 | 2019-05-20 13:25:34,951 [reactorDispatcher-53] execute:87 INFO c.s.c.c.f.AbstractAction - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACKVIEW] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Stack: 1, flow state: IMAGESETUP_STATE, phase: doExec, execution time 1 sec
cloudbreak_1 | 2019-05-20 13:25:34,953 [reactorDispatcher-53] handleStackCreationFailure:208 ERROR c.s.c.c.f.s.p.a.StackCreationService - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACKVIEW] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Error during stack creation flow:
cloudbreak_1 | org.openstack4j.api.exceptions.ServerResponseException: Internal Server Error
cloudbreak_1 | at org.openstack4j.core.transport.HttpExceptionHandler.mapException(HttpExceptionHandler.java:40)
cloudbreak_1 | at org.openstack4j.core.transport.HttpExceptionHandler.mapException(HttpExceptionHandler.java:23)
cloudbreak_1 | at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:51)
cloudbreak_1 | at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:24)
cloudbreak_1 | at org.openstack4j.connectors.jersey2.HttpResponseImpl.getEntity(HttpResponseImpl.java:63)
cloudbreak_1 | at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:225)
cloudbreak_1 | at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:207)
cloudbreak_1 | at org.openstack4j.openstack.image.internal.ImageServiceImpl.list(ImageServiceImpl.java:62)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.handleImageNotFound(OpenStackImageVerifier.java:60)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.getStatusFromImages(OpenStackImageVerifier.java:44)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.getStatus(OpenStackImageVerifier.java:33)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackImageVerifier.exist(OpenStackImageVerifier.java:27)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackSetup.prepareImage(OpenStackSetup.java:50)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler.accept(PrepareImageHandler.java:47)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler.accept(PrepareImageHandler.java:21)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler$$FastClassBySpringCGLIB$$aaf961cd.invoke(<generated>)
cloudbreak_1 | at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
cloudbreak_1 | at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1 | at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1 | at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
cloudbreak_1 | at com.sequenceiq.cloudbreak.cloud.handler.PrepareImageHandler$$EnhancerBySpringCGLIB$$143904db.accept(<generated>)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1 | at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1 | at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1 | at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1 | at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1 | at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1 | at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
cloudbreak_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
cloudbreak_1 | at java.base/java.lang.Thread.run(Thread.java:844)
cloudbreak_1 | 2019-05-20 13:25:35,123 [reactorDispatcher-53] fireCloudbreakEvent:51 INFO c.s.c.s.e.DefaultCloudbreakEventService - [owner:e2c94851-82bb-401a-bdde-65a1382ee5d8] [type:STACKVIEW] [id:1] [name:hdp-test] [flow:2faea964-4555-40a5-88c2-5c273c3ae751] [tracking:fc183563-d1dd-4860-8f75-1c3b0b9b89e7] Firing Cloudbreak event: entityId: 1, type: CREATE_FAILED, message: Infrastructure creation failed. Reason: Internal Server Error



Any help is appreciated.


Cheers,

Martin


4 REPLIES 4

Expert Contributor

Hi @Martin Prodanov,

Looks like the image cannot be automatically imported into your openstack. You can download it and manually import from https://public-repo-1.hortonworks.com/HDP/cloudbreak/cb-hdp-31-1901151759.img

New Contributor

Hi,

thanks for the hint @mmolnar. I tried but it did not help. I still see the same exception. Do you have to specify some particular name in OS for the image?


Cheers,

Martin

Expert Contributor

Hi @Martin Prodanov,

it should be called "cb-hdp-31-1901151759". CB looks for images by name on openstack.

New Contributor

I tried but if fails with the same exception.