Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cloudbreak cluster creation in Openstack failing

avatar
Contributor

I created cluster using hdp-small-default blueprint, I got errors,cluster status show:

Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.

cloudbreak logs :

/cbreak_traefik_1 | time="2017-03-30T04:03:48Z" level=debug msg="Round trip: http://172.17.0.10:3000, code: 200, duration: 2.164172ms tls:version: 303, tls:resume:false, tls:csuite:c02f, tls:server:" 
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] pollWithTimeout:34 WARN  c.s.c.s.PollingService - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Exception occurred in the polling: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.service.cluster.AmbariOperationFailedException: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:55)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:22)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.PollingService.pollWithTimeout(PollingService.java:32)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:60)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:40)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.waitForClusterInstall(AmbariClusterConnector.java:774)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:243)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] pollWithTimeout:39 INFO  c.s.c.s.PollingService - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Polling failure reached the limit which was 5, poller will drop the last exception.
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] handleException:94 ERROR c.s.c.s.c.f.AmbariOperationsStatusCheckerTask - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Ambari operation failed.
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.service.cluster.AmbariOperationFailedException: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:55)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:22)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.PollingService.pollWithTimeout(PollingService.java:32)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:60)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:40)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.waitForClusterInstall(AmbariClusterConnector.java:774)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:243)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] buildAmbariCluster:261 ERROR c.s.c.s.c.f.AmbariClusterConnector - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Error while building the Ambari cluster. Message Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later., throwable: {}
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.core.ClusterException: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.checkPollingResult(AmbariClusterConnector.java:302)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:244)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] accept:70 DEBUG c.s.c.c.f.Flow2Handler - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] flow control event arrived: key: INSTALLCLUSTERFAILED, flowid: 883edc2b-b90c-487f-a195-0438825af8f7, payload: com.sequenceiq.cloudbreak.reactor.api.event.cluster.InstallClusterFailed@1cc7b9f4

I view ambari web UI , I found all the services have installed .

14210-services.png

the HDFS service is not stable, NameNode frequently crashed down and restart .I found namenode out log:

#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError=""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node""
#   Executing /bin/sh -c ""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node""...
ERROR: Could not find a NameNode PID.
ERROR: Could not find a NameNode PID.


so I changed namenode Java heap space from 1GB to 2GB, all services are started successfully .

the cloudbreak ui :

14221-cloudbreak.png

How can I make this cluster show successfully?

any help is appreciated.Thanks!

1 ACCEPTED SOLUTION

avatar
Super Collaborator

@Xu Zhe you can configure the namenode heap space in the blueprint file or you can try to start beefier machines.

If you want to use the blueprint configuration then put this into the blueprint file:

"configurations" : [ { "global" : { "namenode_heapsize" : "1536m", ...  } }, { ... } ]

Br,

R

View solution in original post

4 REPLIES 4

avatar
Super Collaborator

@Xu Zhe you can configure the namenode heap space in the blueprint file or you can try to start beefier machines.

If you want to use the blueprint configuration then put this into the blueprint file:

"configurations" : [ { "global" : { "namenode_heapsize" : "1536m", ...  } }, { ... } ]

Br,

R

avatar
Contributor

Thanks. I go to try

avatar
Contributor

I have successfully installed cloudbreak ,Thank you very much

avatar
Super Collaborator

Please accept then If you find the answer