Reply
New Contributor
Posts: 3
Registered: ‎03-08-2017
Accepted Solution

cloudea cluster bootstrap failed on Google cloud

I was able to start cloudera director successfully.

successfully able to create a cloudera manager instance. Able to login and etc.

When I setup a cluster with 1 master 1 gateway and 3 datanodes. Bootstrap failed. 

Was able to allocate all new instances. At the end of the set up, where it run firstRun it fails with connection refused. Below is the error. 

 

 

[2017-03-07 01:15:53.217 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:15:53.278 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [60, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ...
[2017-03-07 01:15:53.465 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=60, name=CreateHiveDatabase, startTime=Tue Mar 07 01:15:52 UTC 2017, endTime=null, active=true, success=null, resultMessage=null, serviceRef=ApiServiceRef{peerName=null, clusterName=DevCluster, serviceName=CD-HIVE-NJZFPYye}, roleRef=null, hostRef=null, parent=null}
[2017-03-07 01:16:08.525 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Command CreateHiveDatabase with ID 60 completed successfully. Details: ApiCommand{id=60, name=CreateHiveDatabase, startTime=Tue Mar 07 01:15:52 UTC 2017, endTime=Tue Mar 07 01:15:55 UTC 2017, active=false, success=true, resultMessage=Created Hive Metastore Database., serviceRef=ApiServiceRef{peerName=null, clusterName=DevCluster, serviceName=CD-HIVE-NJZFPYye}, roleRef=null, hostRef=null, parent=null}
[2017-03-07 01:16:08.525 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:16:08.563 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> InvokeFirstRunClusterCommandV7/3 [Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredentials{ ...
[2017-03-07 01:16:08.609 +0000] INFO [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Calling firstRun on cluster DevCluster
[2017-03-07 01:16:08.951 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:16:08.960 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for firstRun on cluster DevCluster]
[2017-03-07 01:16:08.961 +0000] INFO [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Waiting for firstRun on cluster DevCluster
[2017-03-07 01:16:08.961 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:16:08.971 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [61, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ...
[2017-03-07 01:16:09.039 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=61, name=First Run, startTime=Tue Mar 07 01:16:08 UTC 2017, endTime=null, active=true, success=null, resultMessage=null, serviceRef=null, roleRef=null, hostRef=null, parent=null}
[2017-03-07 01:16:24.048 +0000] ERROR [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
javax.ws.rs.ProcessingException: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/61: Connection refused (Connection refused)
at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:596)
at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:578)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:748)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:231)
at com.sun.proxy.$Proxy236.readCommand(Unknown Source)
at sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:115)
at com.sun.proxy.$Proxy236.readCommand(Unknown Source)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.pollWhileActive(UnboundedWaitForApiCommand.java:113)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:53)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:33)
at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$9cd7874c.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:104)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:128)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/61: Connection refused (Connection refused)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1377)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1361)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:666)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:746)
... 42 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1181)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:332)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1578)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1607)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1551)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348)
... 48 common frames omitted
[2017-03-07 01:16:34.049 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [61, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ...
[2017-03-07 01:16:34.055 +0000] ERROR [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed

New Contributor
Posts: 3
Registered: ‎03-08-2017

Re: cloudea cluster bootstrap failed on Google cloud

While looking in the logs, I found that it is because the Cloudera Manager is going down with in few minutes of starting. Issues with Service Monitor and Host monitor. Not able to access CM after that. 

 

2017-03-09 03:10:16,919 WARN 1216842385@scm-web-3:com.cloudera.server.web.cmf.StatusProvider: Failed to submit task for getting status from SERVICE_MONITORING
com.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running
at com.cloudera.cmon.MgmtServiceLocator.getNozzleIPC(MgmtServiceLocator.java:145)
at com.cloudera.server.web.cmf.StatusProvider$SubjectStatusCustomFuture.<init>(StatusProvider.java:618)
at com.cloudera.server.web.cmf.StatusProvider.getStatus(StatusProvider.java:1043)
at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getServicesByType(ServiceManagerDaoImpl.java:581)
at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getMgmtService(ServiceManagerDaoImpl.java:608)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:193)
at com.cloudera.api.dao.impl.ManagerDaoBase.access$100(ManagerDaoBase.java:76)
at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:234)
at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:124)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:158)
at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:269)
at com.sun.proxy.$Proxy125.getMgmtService(Unknown Source)
at com.cloudera.api.v1.impl.MgmtServiceResourceImpl.readService(MgmtServiceResourceImpl.java:42)
at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl$RoleConfigGroupsResourceWrapper.<init>(MgmtServiceResourceV3Impl.java:38)
at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl.getRoleConfigGroupsResource(MgmtServiceResourceV3Impl.java:26)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)

New Contributor
Posts: 3
Registered: ‎03-08-2017

Re: cloudea cluster bootstrap failed on Google cloud

I was able to spend more time looking into logs and figured that this is due to services not able to start. CM instance ran out of memory and went to unstable state. 

 

Increasing the memory resolved this issue. 

Posts: 617
Kudos: 70
Solutions: 36
Registered: ‎04-06-2015

Re: cloudea cluster bootstrap failed on Google cloud

Congratulations on solving the issue and thank you for sharing the solution in case it can assist others. :)

 




Cy Jervis, Community Manager - I'm not an expert but will supply relevant content from time to time. :)

Learn more about the Cloudera Community:


Terms of Service


Community Guidelines


How to use the forum

Announcements