Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudbreak cluster creation in Openstack failing

Solved Go to solution

Cloudbreak cluster creation in Openstack failing

New Contributor

I created cluster using hdp-small-default blueprint, I got errors,cluster status show:

Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.

cloudbreak logs :

/cbreak_traefik_1 | time="2017-03-30T04:03:48Z" level=debug msg="Round trip: http://172.17.0.10:3000, code: 200, duration: 2.164172ms tls:version: 303, tls:resume:false, tls:csuite:c02f, tls:server:" 
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] pollWithTimeout:34 WARN  c.s.c.s.PollingService - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Exception occurred in the polling: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.service.cluster.AmbariOperationFailedException: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:55)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:22)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.PollingService.pollWithTimeout(PollingService.java:32)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:60)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:40)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.waitForClusterInstall(AmbariClusterConnector.java:774)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:243)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] pollWithTimeout:39 INFO  c.s.c.s.PollingService - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Polling failure reached the limit which was 5, poller will drop the last exception.
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] handleException:94 ERROR c.s.c.s.c.f.AmbariOperationsStatusCheckerTask - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Ambari operation failed.
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.service.cluster.AmbariOperationFailedException: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:55)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:22)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.PollingService.pollWithTimeout(PollingService.java:32)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:60)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:40)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.waitForClusterInstall(AmbariClusterConnector.java:774)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:243)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] buildAmbariCluster:261 ERROR c.s.c.s.c.f.AmbariClusterConnector - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Error while building the Ambari cluster. Message Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later., throwable: {}
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.core.ClusterException: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.checkPollingResult(AmbariClusterConnector.java:302)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:244)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] accept:70 DEBUG c.s.c.c.f.Flow2Handler - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] flow control event arrived: key: INSTALLCLUSTERFAILED, flowid: 883edc2b-b90c-487f-a195-0438825af8f7, payload: com.sequenceiq.cloudbreak.reactor.api.event.cluster.InstallClusterFailed@1cc7b9f4

I view ambari web UI , I found all the services have installed .

14210-services.png

the HDFS service is not stable, NameNode frequently crashed down and restart .I found namenode out log:

#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError=""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node""
#   Executing /bin/sh -c ""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node""...
ERROR: Could not find a NameNode PID.
ERROR: Could not find a NameNode PID.


so I changed namenode Java heap space from 1GB to 2GB, all services are started successfully .

the cloudbreak ui :

14221-cloudbreak.png

How can I make this cluster show successfully?

any help is appreciated.Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Cloudbreak cluster creation in Openstack failing

Expert Contributor

@Xu Zhe you can configure the namenode heap space in the blueprint file or you can try to start beefier machines.

If you want to use the blueprint configuration then put this into the blueprint file:

"configurations" : [ { "global" : { "namenode_heapsize" : "1536m", ...  } }, { ... } ]

Br,

R

4 REPLIES 4

Re: Cloudbreak cluster creation in Openstack failing

Expert Contributor

@Xu Zhe you can configure the namenode heap space in the blueprint file or you can try to start beefier machines.

If you want to use the blueprint configuration then put this into the blueprint file:

"configurations" : [ { "global" : { "namenode_heapsize" : "1536m", ...  } }, { ... } ]

Br,

R

Re: Cloudbreak cluster creation in Openstack failing

New Contributor

Thanks. I go to try

Re: Cloudbreak cluster creation in Openstack failing

New Contributor

I have successfully installed cloudbreak ,Thank you very much

Re: Cloudbreak cluster creation in Openstack failing

Expert Contributor

Please accept then If you find the answer

Don't have an account?
Coming from Hortonworks? Activate your account here