Infrastructure creation failed. Reason: null - CBD 1.6.3

[#1]

Hey everyone,

I am currently trying to deploy an Hadoop Cluster using Cloudbreak 1.6.3 with the blueprint hdp-small-default over Openstack Ocata. I'm working behind a Proxy.

I got an issue : Infrastructure creation failed. Reason: null

I don't use HEAT so i'm using the NATIVE option to deploy HDP Cluster.

Any idea how could I fix that ?

Thank you very much!

[#2]

@faraon clément

Please note, that Ocata is not supported officially by Cloudbreak yet, you can check the documentation.

It might work but afaik it was not tested against it. It would be the most useful if you could send the Cloudbreak logs (either from your cbd directory or generated by "cbd logs cloudbreak"), it should contain the root cause of the error.

Hope this helps!

[#3]

@pdarvasi

tank you very much for your answer. I have seen this documentation so I can understand that some problems happened ...

I guess that the problem occurs just after credentials creation but I don't understand very well the reason ...

I give you logs details :

cloudbreak_1   | 2017-03-23 08:43:40,379 [reactorDispatcher-36] buildLogContextForReactorHandler:48 INFO  c.s.c.l.LogContextAspects -
 [owner:1013d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] A Reactor event handler's 'accept' method has been inter
cepted: execution(Consumer.accept(..)), MDC logger context is built.

cloudbreak_1   | 2017-03-23 08:43:40,379 [reactorDispatcher-36] accept:56 INFO  c.s.c.c.h.LaunchStackHandler - [owner:1013d790-f4ad-4
fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Received event: Event{id=null, headers={FLOW_CHAIN_ID=561f3d40-9e74-4512-a8ee-0
90a6c444efe, FLOW_ID=4590e3de-d876-45d3-bed7-ea9bb5d83942}, replyTo=null, key=LAUNCHSTACKREQUEST, data=CloudStackRequest{, cloudStack=CloudS
tack{groups=[com.sequenceiq.cloudbreak.cloud.model.Group@626305be, com.sequenceiq.cloudbreak.cloud.model.Group@6542b76a, com.sequenceiq.clou
dbreak.cloud.model.Group@79237cb3, com.sequenceiq.cloudbreak.cloud.model.Group@5f42ace4, com.sequenceiq.cloudbreak.cloud.model.Group@25f61c4
7], network=com.sequenceiq.cloudbreak.cloud.model.Network@557f1674, image=com.sequenceiq.cloudbreak.cloud.model.Image@375b2e8c}}}

cloudbreak_1   | 2017-03-23 08:43:40,379 [reactorDispatcher-36] authenticate:37 INFO  c.s.c.c.o.a.OpenStackAuthenticator - [owner:101
3d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Authenticating to openstack ...

cloudbreak_1   | 2017-03-23 08:43:40,703 [reactorDispatcher-36] create:45 INFO  c.s.c.c.o.a.OpenStackCredentialConnector - [owner:101
3d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Create credential: com.sequenceiq.cloudbreak.cloud.model.CloudCrede
ntial@7d34810f

cloudbreak_1   | 2017-03-23 08:43:41,105 [reactorDispatcher-36] create:58 INFO  c.s.c.c.o.a.OpenStackCredentialConnector - [owner:101
3d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Credential has been created: com.sequenceiq.cloudbreak.cloud.model.
CloudCredential@7d34810f, kp: NovaKeypair{name=cb-testluter_10-cluser-2, public_key=ssh-rsa AAAAB****Hl2Eq0b Generated-by-Nova, fingerprint=08:a3:d5:1a:ea:59:17:e9:17:85:ba:a5:e6:da:c5:97}

cloudbreak_1   | 2017-03-23 08:43:41,106 [reactorDispatcher-36] accept:70 DEBUG c.s.c.c.f.Flow2Handler - [owner:1013d790-f4ad-4fe3-b1
04-dbea54b31e71] [type:STACK] [id:10] [name:testluter] flow control event arrived: key: LAUNCHSTACKRESULT_ERROR, flowid: 4590e3de-d876-45d3-
bed7-ea9bb5d83942, payload: LaunchStackResult{status=FAILED, statusReason='', errorDetails=java.lang.NullPointerException, request=CloudStac
kRequest{, cloudStack=CloudStack{groups=[com.sequenceiq.cloudbreak.cloud.model.Group@626305be, com.sequenceiq.cloudbreak.cloud.model.Group@6
542b76a, com.sequenceiq.cloudbreak.cloud.model.Group@79237cb3, com.sequenceiq.cloudbreak.cloud.model.Group@5f42ace4, com.sequenceiq.cloudbre
ak.cloud.model.Group@25f61c47], network=com.sequenceiq.cloudbreak.cloud.model.Network@557f1674, image=com.sequenceiq.cloudbreak.cloud.model.
Image@375b2e8c}}, results=null}

cloudbreak_1   | 2017-03-23 08:43:41,159 [reactorDispatcher-36] execute:64 INFO  c.s.c.c.f.AbstractAction - [owner:1013d790-f4ad-4fe3
-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Stack: 10, flow state: START_PROVISIONING_STATE, phase: service, execution time 0 
sec

cloudbreak_1   | 2017-03-23 08:43:41,159 [reactorDispatcher-36] handleStackCreationFailure:176 ERROR c.s.c.c.f.s.p.a.StackCreationSer
vice - [owner:1013d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Error during stack creation flow:
cloudbreak_1   | java.lang.NullPointerException: null
cloudbreak_1   |     at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:210)
cloudbreak_1   |     at org.openstack4j.openstack.identity.internal.TenantServiceImpl.getByName(TenantServiceImpl.java:34)
cloudbreak_1   |     at com.sequenceiq.cloudbreak.cloud.openstack.nativ.context.OpenStackContextBuilder.contextInit(OpenStackCont
extBuilder.java:46)
cloudbreak_1   |     at com.sequenceiq.cloudbreak.cloud.openstack.nativ.context.OpenStackContextBuilder.contextInit(OpenStackCont
extBuilder.java:27)
cloudbreak_1   |     at com.sequenceiq.cloudbreak.cloud.template.AbstractResourceConnector.launch(AbstractResourceConnector.java:
55)
cloudbreak_1   |     at com.sequenceiq.cloudbreak.cloud.handler.LaunchStackHandler.accept(LaunchStackHandler.java:69)
cloudbreak_1   |     at com.sequenceiq.cloudbreak.cloud.handler.LaunchStackHandler.accept(LaunchStackHandler.java:33)
cloudbreak_1   |     at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)
cloudbreak_1   |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
cloudbreak_1   |     at java.lang.reflect.Method.invoke(Method.java:498)
cloudbreak_1   |     at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
cloudbreak_1   |     at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.j
ava:190)
cloudbreak_1   |     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
cloudbreak_1   |     at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceIntercep
tor.java:52)
cloudbreak_1   |     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1   |     at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:9
2)
cloudbreak_1   |     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
cloudbreak_1   |     at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
cloudbreak_1   |     at com.sun.proxy.$Proxy209.accept(Unknown Source)
cloudbreak_1   |     at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1   |     at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1   |     at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1   |     at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1   |     at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1   |     at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1   |     at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1   |     at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1   |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
cloudbreak_1   |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
cloudbreak_1   |     at java.lang.Thread.run(Thread.java:745)
cloudbreak_1   | 2017-03-23 08:43:41,160 [reactorDispatcher-36] fireEventAndLog:24 DEBUG c.s.c.c.f.s.FlowMessageService - [owner:1013
d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] STACK_INFRASTRUCTURE_CREATE_FAILED [STACK_FLOW_STEP].
cloudbreak_1   | 2017-03-23 08:43:41,160 [reactorDispatcher-36] fireCloudbreakEvent:51 INFO  c.s.c.s.e.DefaultCloudbreakEventService 
- [owner:1013d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Firing Cloudbreak event: CloudbreakEventData{entityId=1
0, eventType='UPDATE_IN_PROGRESS', eventMessage='Infrastructure creation failed. Reason: null'}

cloudbreak_1   | 2017-03-23 08:43:41,160 [reactorDispatcher-36] handleFailure:212 DEBUG c.s.c.c.f.s.p.a.StackCreationService - [owner
:1013d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Nothing to do. OnFailureAction DO_NOTHING

cloudbreak_1   | 2017-03-23 08:43:41,161 [reactorDispatcher-46] accept:33 INFO  c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] 
[type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Handling cloudbreak event: Event{id=null, headers=null, replyTo=null, key=CLOUDBR
EAK_EVENT, data=CloudbreakEventData{entityId=10, eventType='UPDATE_IN_PROGRESS', eventMessage='Infrastructure creation failed. Reason: null'
}}

cloudbreak_1   | 2017-03-23 08:43:41,161 [reactorDispatcher-46] accept:36 INFO  c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] 
[type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Persisting data: CloudbreakEventData{entityId=10, eventType='UPDATE_IN_PROGRESS',
 eventMessage='Infrastructure creation failed. Reason: null'}

cloudbreak_1   | 2017-03-23 08:43:41,162 [reactorDispatcher-46] createStackEvent:66 DEBUG c.s.c.s.e.DefaultCloudbreakEventService - [
owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Creating stack event from: CloudbreakEventData{entityId=10, even
tType='UPDATE_IN_PROGRESS', eventMessage='Infrastructure creation failed. Reason: null'}

cloudbreak_1   | 2017-03-23 08:43:41,207 [reactorDispatcher-46] createStackEvent:71 INFO  c.s.c.s.e.DefaultCloudbreakEventService - [
owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Created stack event: CloudbreakEvent{id=50, eventType='UPDATE_IN
_PROGRESS', eventTimestamp=Thu Mar 23 08:43:41 UTC 2017, eventMessage='Infrastructure creation failed. Reason: null', owner='1013d790-f4ad-4
fe3-b104-dbea54b31e71', account='seq1234567', cloud='OPENSTACK', region='RegionOne', blueprintName='hdp-small-default', instanceGroup='null'
, blueprintId=1', stackId=10', clusterId=6', clusterName=testluter}

cloudbreak_1   | 2017-03-23 08:43:41,208 [reactorDispatcher-46] accept:38 INFO  c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] 
[type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Sending notification with data: CloudbreakEvent{id=50, eventType='UPDATE_IN_PROGR
ESS', eventTimestamp=Thu Mar 23 08:43:41 UTC 2017, eventMessage='Infrastructure creation failed. Reason: null', owner='1013d790-f4ad-4fe3-b1
04-dbea54b31e71', account='seq1234567', cloud='OPENSTACK', region='RegionOne', blueprintName='hdp-small-default', instanceGroup='null', blue
printId=1', stackId=10', clusterId=6', clusterName=testluter}

cloudbreak_1   | 2017-03-23 08:43:41,241 [reactorDispatcher-36] fireEventAndLog:24 DEBUG c.s.c.c.f.s.FlowMessageService - [owner:1013
d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] STACK_INFRASTRUCTURE_CREATE_FAILED [STACK_FLOW_STEP].

cloudbreak_1   | 2017-03-23 08:43:41,241 [reactorDispatcher-36] fireCloudbreakEvent:51 INFO  c.s.c.s.e.DefaultCloudbreakEventService 
- [owner:1013d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Firing Cloudbreak event: CloudbreakEventData{entityId=1
0, eventType='CREATE_FAILED', eventMessage='Infrastructure creation failed. Reason: null'}

cloudbreak_1   | 2017-03-23 08:43:41,241 [reactorDispatcher-36] sendEvent:103 INFO  c.s.c.c.f.AbstractAction - [owner:1013d790-f4ad-4
fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Triggering event: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@10ca94
6f

cloudbreak_1   | 2017-03-23 08:43:41,241 [reactorDispatcher-36] stateChanged:125 INFO  c.s.c.c.f.c.AbstractFlowConfiguration - [owner
:1013d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter]  changed from ObjectState [getIds()=[START_PROVISIONING_STATE],
 getClass()=class org.springframework.statemachine.state.ObjectState, hashCode()=1563669019, toString()=AbstractState [id=START_PROVISIONING
_STATE, pseudoState=null, deferred=null, entryActions=[com.sequenceiq.cloudbreak.core.flow2.stack.provision.action.StackCreationActions$3@37
257c66], exitActions=null, regions=[], submachine=null]] to ObjectState [getIds()=[STACK_CREATION_FAILED_STATE], getClass()=class org.spring
framework.statemachine.state.ObjectState, hashCode()=1418289361, toString()=AbstractState [id=STACK_CREATION_FAILED_STATE, pseudoState=null,
 deferred=null, entryActions=[com.sequenceiq.cloudbreak.core.flow2.stack.provision.action.StackCreationActions$8@227aa096], exitActions=null
, regions=[], submachine=null]]
cloudbreak_1   | 2017-03-23 08:43:41,242 [reactorDispatcher-36] accept:33 INFO  c.s.c.s.e.CloudbreakEventHandler - [owner:1013d790-f4
ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:10] [name:testluter] Handling cloudbreak event: Event{id=null, headers=null, replyTo=null, key=C
LOUDBREAK_EVENT, data=CloudbreakEventData{entityId=10, eventType='CREATE_FAILED', eventMessage='Infrastructure creation failed. Reason: null
'}}

cloudbreak_1   | 2017-03-23 08:43:41,242 [reactorDispatcher-36] accept:36 INFO  c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] 
[type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Persisting data: CloudbreakEventData{entityId=10, eventType='CREATE_FAILED', even
tMessage='Infrastructure creation failed. Reason: null'}

cloudbreak_1   | 2017-03-23 08:43:41,242 [reactorDispatcher-36] createStackEvent:66 DEBUG c.s.c.s.e.DefaultCloudbreakEventService - [
owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Creating stack event from: CloudbreakEventData{entityId=10, even
tType='CREATE_FAILED', eventMessage='Infrastructure creation failed. Reason: null'}

cloudbreak_1   | 2017-03-23 08:43:41,242 [reactorDispatcher-45] accept:70 DEBUG c.s.c.c.f.Flow2Handler - [owner:1013d790-f4ad-4fe3-b1
04-dbea54b31e71] [type:STACK] [id:7] [name:clusteressai2] flow control event arrived: key: STACK_CREATION_FAILHANDLED, flowid: 4590e3de-d876
-45d3-bed7-ea9bb5d83942, payload: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@10ca946f

cloudbreak_1   | 2017-03-23 08:43:41,262 [reactorDispatcher-45] execute:64 INFO  c.s.c.c.f.AbstractAction - [owner:1013d790-f4ad-4fe3
-b104-dbea54b31e71] [type:STACK] [id:7] [name:clusteressai2] Stack: 10, flow state: STACK_CREATION_FAILED_STATE, phase: service, execution t
ime 0 sec

cloudbreak_1   | 2017-03-23 08:43:41,262 [reactorDispatcher-45] sendEvent:103 INFO  c.s.c.c.f.AbstractAction - [owner:1013d790-f4ad-4
fe3-b104-dbea54b31e71] [type:STACK] [id:7] [name:clusteressai2] Triggering event: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@10c
a946f

cloudbreak_1   | 2017-03-23 08:43:41,262 [reactorDispatcher-45] stateChanged:125 INFO  c.s.c.c.f.c.AbstractFlowConfiguration - [owner
:1013d790-f4ad-4fe3-b104-dbea54b31e71] [type:STACK] [id:7] [name:clusteressai2]  changed from ObjectState [getIds()=[STACK_CREATION_FAILED_S
TATE], getClass()=class org.springframework.statemachine.state.ObjectState, hashCode()=1418289361, toString()=AbstractState [id=STACK_CREATI
ON_FAILED_STATE, pseudoState=null, deferred=null, entryActions=[com.sequenceiq.cloudbreak.core.flow2.stack.provision.action.StackCreationAct
ions$8@227aa096], exitActions=null, regions=[], submachine=null]] to ObjectState [getIds()=[FINAL_STATE], getClass()=class org.springframewo
rk.statemachine.state.ObjectState, hashCode()=158514087, toString()=AbstractState [id=FINAL_STATE, pseudoState=org.springframework.statemach
ine.state.DefaultPseudoState@7bcf0eef, deferred=null, entryActions=[com.sequenceiq.cloudbreak.core.flow2.FlowFinalizeAction@7ad046de], exitA
ctions=null, regions=[], submachine=null]]
cloudbreak_1   | 2017-03-23 08:43:41,263 [reactorDispatcher-45] stop:127 INFO  o.s.s.s.LifecycleObjectSupport - [owner:1013d790-f4ad-
4fe3-b104-dbea54b31e71] [type:STACK] [id:7] [name:clusteressai2] stopped org.springframework.statemachine.support.DefaultStateMachineExecuto
r@3717503e


cloudbreak_1   | 2017-03-23 08:43:41,263 [reactorDispatcher-45] stop:127 INFO  o.s.s.s.LifecycleObjectSupport - [owner:1013d790-f4ad-
4fe3-b104-dbea54b31e71] [type:STACK] [id:7] [name:clusteressai2] stopped STACK_CREATION_FINISHED_STATE START_PROVISIONING_STATE INIT_STATE S
TACK_CREATION_FAILED_STATE TLS_SETUP_STATE PROVISIONING_FINISHED_STATE FINAL_STATE IMAGE_CHECK_STATE IMAGESETUP_STATE COLLECTMETADATA_STATE 
SETUP_STATE  /  / id=3d058f95-50d0-41b2-b1eb-5d07c6e7961b

cloudbreak_1   | 2017-03-23 08:43:41,263 [reactorDispatcher-45] finalizeFlow:95 DEBUG c.s.c.c.f.Flow2Handler - [owner:1013d790-f4ad-4
fe3-b104-dbea54b31e71] [type:STACK] [id:7] [name:clusteressai2] flow finalizing arrived: id: 4590e3de-d876-45d3-bed7-ea9bb5d83942

cloudbreak_1   | 2017-03-23 08:43:41,281 [reactorDispatcher-36] createStackEvent:71 INFO  c.s.c.s.e.DefaultCloudbreakEventService - [
owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Created stack event: CloudbreakEvent{id=51, eventType='CREATE_FA
ILED', eventTimestamp=Thu Mar 23 08:43:41 UTC 2017, eventMessage='Infrastructure creation failed. Reason: null', owner='1013d790-f4ad-4fe3-b
104-dbea54b31e71', account='seq1234567', cloud='OPENSTACK', region='RegionOne', blueprintName='hdp-small-default', instanceGroup='null', blu
eprintId=1', stackId=10', clusterId=6', clusterName=testluter}

cloudbreak_1   | 2017-03-23 08:43:41,281 [reactorDispatcher-36] accept:38 INFO  c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] 
[type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Sending notification with data: CloudbreakEvent{id=51, eventType='CREATE_FAILED',
 eventTimestamp=Thu Mar 23 08:43:41 UTC 2017, eventMessage='Infrastructure creation failed. Reason: null', owner='1013d790-f4ad-4fe3-b104-db
ea54b31e71', account='seq1234567', cloud='OPENSTACK', region='RegionOne', blueprintName='hdp-small-default', instanceGroup='null', blueprint
Id=1', stackId=10', clusterId=6', clusterName=testluter}

[#4 · Accepted solution]

@faraon clément

It turned out that you have found an issue, so I am filing a Jira ticket for that.

As a workaround, you might try to get it working with HEAT (or even with Keystone V2), it should work.

Hope this helps!