cloudea cluster bootstrap failed on Google cloud

clouderaGCP — Fri, 16 Sep 2022 15:43:30 GMT

I was able to start cloudera director successfully.

successfully able to create a cloudera manager instance. Able to login and etc.

When I setup a cluster with 1 master 1 gateway and 3 datanodes. Bootstrap failed.

Was able to allocate all new instances. At the end of the set up, where it run firstRun it fails with connection refused. Below is the error.

[2017-03-07 01:15:53.217 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:15:53.278 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [60, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ...
[2017-03-07 01:15:53.465 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=60, name=CreateHiveDatabase, startTime=Tue Mar 07 01:15:52 UTC 2017, endTime=null, active=true, success=null, resultMessage=null, serviceRef=ApiServiceRef{peerName=null, clusterName=DevCluster, serviceName=CD-HIVE-NJZFPYye}, roleRef=null, hostRef=null, parent=null}
[2017-03-07 01:16:08.525 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Command CreateHiveDatabase with ID 60 completed successfully. Details: ApiCommand{id=60, name=CreateHiveDatabase, startTime=Tue Mar 07 01:15:52 UTC 2017, endTime=Tue Mar 07 01:15:55 UTC 2017, active=false, success=true, resultMessage=Created Hive Metastore Database., serviceRef=ApiServiceRef{peerName=null, clusterName=DevCluster, serviceName=CD-HIVE-NJZFPYye}, roleRef=null, hostRef=null, parent=null}
[2017-03-07 01:16:08.525 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:16:08.563 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> InvokeFirstRunClusterCommandV7/3 [Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredentials{ ...
[2017-03-07 01:16:08.609 +0000] INFO [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Calling firstRun on cluster DevCluster
[2017-03-07 01:16:08.951 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:16:08.960 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for firstRun on cluster DevCluster]
[2017-03-07 01:16:08.961 +0000] INFO [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Waiting for firstRun on cluster DevCluster
[2017-03-07 01:16:08.961 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:16:08.971 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [61, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ...
[2017-03-07 01:16:09.039 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=61, name=First Run, startTime=Tue Mar 07 01:16:08 UTC 2017, endTime=null, active=true, success=null, resultMessage=null, serviceRef=null, roleRef=null, hostRef=null, parent=null}
[2017-03-07 01:16:24.048 +0000] ERROR [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
javax.ws.rs.ProcessingException: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/61: Connection refused (Connection refused)
at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:596)
at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:578)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:748)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:231)
at com.sun.proxy.$Proxy236.readCommand(Unknown Source)
at sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:115)
at com.sun.proxy.$Proxy236.readCommand(Unknown Source)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.pollWhileActive(UnboundedWaitForApiCommand.java:113)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:53)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:33)
at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$9cd7874c.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:104)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:128)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/61: Connection refused (Connection refused)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1377)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1361)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:666)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:746)
... 42 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1181)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:332)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1578)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1607)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1551)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348)
... 48 common frames omitted
[2017-03-07 01:16:34.049 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [61, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ...
[2017-03-07 01:16:34.055 +0000] ERROR [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed

Re: cloudea cluster bootstrap failed on Google cloud

clouderaGCP — Thu, 09 Mar 2017 03:46:47 GMT

While looking in the logs, I found that it is because the Cloudera Manager is going down with in few minutes of starting. Issues with Service Monitor and Host monitor. Not able to access CM after that.

2017-03-09 03:10:16,919 WARN 1216842385@scm-web-3:com.cloudera.server.web.cmf.StatusProvider: Failed to submit task for getting status from SERVICE_MONITORING
com.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running
at com.cloudera.cmon.MgmtServiceLocator.getNozzleIPC(MgmtServiceLocator.java:145)
at com.cloudera.server.web.cmf.StatusProvider$SubjectStatusCustomFuture.<init>(StatusProvider.java:618)
at com.cloudera.server.web.cmf.StatusProvider.getStatus(StatusProvider.java:1043)
at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getServicesByType(ServiceManagerDaoImpl.java:581)
at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getMgmtService(ServiceManagerDaoImpl.java:608)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:193)
at com.cloudera.api.dao.impl.ManagerDaoBase.access$100(ManagerDaoBase.java:76)
at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:234)
at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:124)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:158)
at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:269)
at com.sun.proxy.$Proxy125.getMgmtService(Unknown Source)
at com.cloudera.api.v1.impl.MgmtServiceResourceImpl.readService(MgmtServiceResourceImpl.java:42)
at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl$RoleConfigGroupsResourceWrapper.<init>(MgmtServiceResourceV3Impl.java:38)
at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl.getRoleConfigGroupsResource(MgmtServiceResourceV3Impl.java:26)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)

Re: cloudea cluster bootstrap failed on Google cloud

clouderaGCP — Thu, 09 Mar 2017 05:27:12 GMT

I was able to spend more time looking into logs and figured that this is due to services not able to start. CM instance ran out of memory and went to unstable state.

Increasing the memory resolved this issue.

Re: cloudea cluster bootstrap failed on Google cloud

cjervis — Thu, 09 Mar 2017 13:46:43 GMT

Congratulations on solving the issue and thank you for sharing the solution in case it can assist others. 🙂

question Re: cloudea cluster bootstrap failed on Google cloud in Support Questions

cloudea cluster bootstrap failed on Google cloud

Re: cloudea cluster bootstrap failed on Google cloud

Re: cloudea cluster bootstrap failed on Google cloud

Re: cloudea cluster bootstrap failed on Google cloud