Cloudbreak cluster creation in Openstack failing

[#1]

I created cluster using hdp-small-default blueprint, I got errors,cluster status show:

Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.

cloudbreak logs :

/cbreak_traefik_1 | time="2017-03-30T04:03:48Z" level=debug msg="Round trip: http://172.17.0.10:3000, code: 200, duration: 2.164172ms tls:version: 303, tls:resume:false, tls:csuite:c02f, tls:server:" 
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] pollWithTimeout:34 WARN  c.s.c.s.PollingService - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Exception occurred in the polling: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.service.cluster.AmbariOperationFailedException: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:55)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:22)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.PollingService.pollWithTimeout(PollingService.java:32)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:60)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:40)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.waitForClusterInstall(AmbariClusterConnector.java:774)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:243)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] pollWithTimeout:39 INFO  c.s.c.s.PollingService - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Polling failure reached the limit which was 5, poller will drop the last exception.
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] handleException:94 ERROR c.s.c.s.c.f.AmbariOperationsStatusCheckerTask - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Ambari operation failed.
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.service.cluster.AmbariOperationFailedException: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:55)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:22)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.PollingService.pollWithTimeout(PollingService.java:32)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:60)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:40)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.waitForClusterInstall(AmbariClusterConnector.java:774)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:243)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] buildAmbariCluster:261 ERROR c.s.c.s.c.f.AmbariClusterConnector - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Error while building the Ambari cluster. Message Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later., throwable: {}
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.core.ClusterException: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.checkPollingResult(AmbariClusterConnector.java:302)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:244)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] accept:70 DEBUG c.s.c.c.f.Flow2Handler - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] flow control event arrived: key: INSTALLCLUSTERFAILED, flowid: 883edc2b-b90c-487f-a195-0438825af8f7, payload: com.sequenceiq.cloudbreak.reactor.api.event.cluster.InstallClusterFailed@1cc7b9f4

I view ambari web UI , I found all the services have installed .

the HDFS service is not stable, NameNode frequently crashed down and restart .I found namenode out log:

#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError=""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node""
#   Executing /bin/sh -c ""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node""...
ERROR: Could not find a NameNode PID.
ERROR: Could not find a NameNode PID.

so I changed namenode Java heap space from 1GB to 2GB, all services are started successfully .

the cloudbreak ui :

How can I make this cluster show successfully?

any help is appreciated.Thanks!

[#2 · Accepted solution]

@Xu Zhe you can configure the namenode heap space in the blueprint file or you can try to start beefier machines.

If you want to use the blueprint configuration then put this into the blueprint file:

"configurations" : [ { "global" : { "namenode_heapsize" : "1536m", ...  } }, { ... } ]

Br,

R

[#3]

Thanks. I go to try

[#4]

I have successfully installed cloudbreak ,Thank you very much

[#5]

Please accept then If you find the answer