Cloudbreak cluster creation in Openstack failing

geniuszhe — Sun, 18 Aug 2019 09:11:16 GMT

I created cluster using hdp-small-default blueprint, I got errors,cluster status show:

Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.

cloudbreak logs :

/cbreak_traefik_1 | time="2017-03-30T04:03:48Z" level=debug msg="Round trip: http://172.17.0.10:3000, code: 200, duration: 2.164172ms tls:version: 303, tls:resume:false, tls:csuite:c02f, tls:server:" 
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] pollWithTimeout:34 WARN  c.s.c.s.PollingService - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Exception occurred in the polling: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.service.cluster.AmbariOperationFailedException: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:55)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:22)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.PollingService.pollWithTimeout(PollingService.java:32)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:60)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:40)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.waitForClusterInstall(AmbariClusterConnector.java:774)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:243)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] pollWithTimeout:39 INFO  c.s.c.s.PollingService - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Polling failure reached the limit which was 5, poller will drop the last exception.
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] handleException:94 ERROR c.s.c.s.c.f.AmbariOperationsStatusCheckerTask - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Ambari operation failed.
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.service.cluster.AmbariOperationFailedException: Ambari operation failed: [component: 'CLUSTER_INSTALL', requestID: '1']
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:55)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationsStatusCheckerTask.checkStatus(AmbariOperationsStatusCheckerTask.java:22)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.PollingService.pollWithTimeout(PollingService.java:32)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:60)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariOperationService.waitForOperations(AmbariOperationService.java:40)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.waitForClusterInstall(AmbariClusterConnector.java:774)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:243)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] buildAmbariCluster:261 ERROR c.s.c.s.c.f.AmbariClusterConnector - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] Error while building the Ambari cluster. Message Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later., throwable: {}
/cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.core.ClusterException: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.checkPollingResult(AmbariClusterConnector.java:302)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.service.cluster.flow.AmbariClusterConnector.buildAmbariCluster(AmbariClusterConnector.java:244)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.core.cluster.AmbariClusterCreationService.buildAmbariCluster(AmbariClusterCreationService.java:27)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:35)
/cbreak_cloudbreak_1 |  at com.sequenceiq.cloudbreak.reactor.handler.cluster.InstallClusterHandler.accept(InstallClusterHandler.java:18)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:317)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus$3.accept(EventBus.java:310)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
/cbreak_cloudbreak_1 |  at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:591)
/cbreak_cloudbreak_1 |  at reactor.bus.EventBus.accept(EventBus.java:63)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
/cbreak_cloudbreak_1 |  at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
/cbreak_cloudbreak_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
/cbreak_cloudbreak_1 |  at java.lang.Thread.run(Thread.java:745)
/cbreak_cloudbreak_1 | 2017-03-30 04:03:48,647 [reactorDispatcher-98] accept:70 DEBUG c.s.c.c.f.Flow2Handler - [owner:ae2e5c22-25b0-4283-aeb3-dcc06aa706ab] [type:STACK] [id:29] [name:bigdata2] flow control event arrived: key: INSTALLCLUSTERFAILED, flowid: 883edc2b-b90c-487f-a195-0438825af8f7, payload: com.sequenceiq.cloudbreak.reactor.api.event.cluster.InstallClusterFailed@1cc7b9f4

I view ambari web UI , I found all the services have installed .

the HDFS service is not stable, NameNode frequently crashed down and restart .I found namenode out log:

#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError=""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node""
#   Executing /bin/sh -c ""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node""...
ERROR: Could not find a NameNode PID.
ERROR: Could not find a NameNode PID.

so I changed namenode Java heap space from 1GB to 2GB, all services are started successfully .

the cloudbreak ui :

How can I make this cluster show successfully?

any help is appreciated.Thanks!

Re: Cloudbreak cluster creation in Openstack failing

rdoktorics — Thu, 30 Mar 2017 12:54:57 GMT

@Xu Zhe you can configure the namenode heap space in the blueprint file or you can try to start beefier machines.

If you want to use the blueprint configuration then put this into the blueprint file:

"configurations" : [ { "global" : { "namenode_heapsize" : "1536m", ...  } }, { ... } ]

Br,

Re: Cloudbreak cluster creation in Openstack failing

geniuszhe — Thu, 30 Mar 2017 13:05:16 GMT

Thanks. I go to try

Re: Cloudbreak cluster creation in Openstack failing

geniuszhe — Thu, 30 Mar 2017 14:05:12 GMT

I have successfully installed cloudbreak ,Thank you very much

Re: Cloudbreak cluster creation in Openstack failing

rdoktorics — Thu, 30 Mar 2017 15:12:35 GMT

Please accept then If you find the answer

question Cloudbreak cluster creation in Openstack failing in Archives of Support Questions (Read Only)

Cloudbreak cluster creation in Openstack failing

Re: Cloudbreak cluster creation in Openstack failing

Re: Cloudbreak cluster creation in Openstack failing

Re: Cloudbreak cluster creation in Openstack failing

Re: Cloudbreak cluster creation in Openstack failing