Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

cloudea cluster bootstrap failed on Google cloud

avatar
New Contributor

I was able to start cloudera director successfully.

successfully able to create a cloudera manager instance. Able to login and etc.

When I setup a cluster with 1 master 1 gateway and 3 datanodes. Bootstrap failed. 

Was able to allocate all new instances. At the end of the set up, where it run firstRun it fails with connection refused. Below is the error. 

 

 

[2017-03-07 01:15:53.217 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:15:53.278 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [60, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ...
[2017-03-07 01:15:53.465 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=60, name=CreateHiveDatabase, startTime=Tue Mar 07 01:15:52 UTC 2017, endTime=null, active=true, success=null, resultMessage=null, serviceRef=ApiServiceRef{peerName=null, clusterName=DevCluster, serviceName=CD-HIVE-NJZFPYye}, roleRef=null, hostRef=null, parent=null}
[2017-03-07 01:16:08.525 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Command CreateHiveDatabase with ID 60 completed successfully. Details: ApiCommand{id=60, name=CreateHiveDatabase, startTime=Tue Mar 07 01:15:52 UTC 2017, endTime=Tue Mar 07 01:15:55 UTC 2017, active=false, success=true, resultMessage=Created Hive Metastore Database., serviceRef=ApiServiceRef{peerName=null, clusterName=DevCluster, serviceName=CD-HIVE-NJZFPYye}, roleRef=null, hostRef=null, parent=null}
[2017-03-07 01:16:08.525 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:16:08.563 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> InvokeFirstRunClusterCommandV7/3 [Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredentials{ ...
[2017-03-07 01:16:08.609 +0000] INFO [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Calling firstRun on cluster DevCluster
[2017-03-07 01:16:08.951 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:16:08.960 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for firstRun on cluster DevCluster]
[2017-03-07 01:16:08.961 +0000] INFO [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Waiting for firstRun on cluster DevCluster
[2017-03-07 01:16:08.961 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{}
[2017-03-07 01:16:08.971 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [61, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ...
[2017-03-07 01:16:09.039 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=61, name=First Run, startTime=Tue Mar 07 01:16:08 UTC 2017, endTime=null, active=true, success=null, resultMessage=null, serviceRef=null, roleRef=null, hostRef=null, parent=null}
[2017-03-07 01:16:24.048 +0000] ERROR [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
javax.ws.rs.ProcessingException: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/61: Connection refused (Connection refused)
at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:596)
at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:578)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:748)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:231)
at com.sun.proxy.$Proxy236.readCommand(Unknown Source)
at sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:115)
at com.sun.proxy.$Proxy236.readCommand(Unknown Source)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.pollWhileActive(UnboundedWaitForApiCommand.java:113)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:53)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:33)
at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$9cd7874c.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:104)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:128)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/61: Connection refused (Connection refused)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1377)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1361)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:666)
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:746)
... 42 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1181)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:332)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1578)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1607)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1551)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348)
... 48 common frames omitted
[2017-03-07 01:16:34.049 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [61, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ...
[2017-03-07 01:16:34.055 +0000] ERROR [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed

1 ACCEPTED SOLUTION

avatar
New Contributor

I was able to spend more time looking into logs and figured that this is due to services not able to start. CM instance ran out of memory and went to unstable state. 

 

Increasing the memory resolved this issue. 

View solution in original post

3 REPLIES 3

avatar
New Contributor

While looking in the logs, I found that it is because the Cloudera Manager is going down with in few minutes of starting. Issues with Service Monitor and Host monitor. Not able to access CM after that. 

 

2017-03-09 03:10:16,919 WARN 1216842385@scm-web-3:com.cloudera.server.web.cmf.StatusProvider: Failed to submit task for getting status from SERVICE_MONITORING
com.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running
at com.cloudera.cmon.MgmtServiceLocator.getNozzleIPC(MgmtServiceLocator.java:145)
at com.cloudera.server.web.cmf.StatusProvider$SubjectStatusCustomFuture.<init>(StatusProvider.java:618)
at com.cloudera.server.web.cmf.StatusProvider.getStatus(StatusProvider.java:1043)
at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getServicesByType(ServiceManagerDaoImpl.java:581)
at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getMgmtService(ServiceManagerDaoImpl.java:608)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:193)
at com.cloudera.api.dao.impl.ManagerDaoBase.access$100(ManagerDaoBase.java:76)
at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:234)
at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:124)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:158)
at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:269)
at com.sun.proxy.$Proxy125.getMgmtService(Unknown Source)
at com.cloudera.api.v1.impl.MgmtServiceResourceImpl.readService(MgmtServiceResourceImpl.java:42)
at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl$RoleConfigGroupsResourceWrapper.<init>(MgmtServiceResourceV3Impl.java:38)
at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl.getRoleConfigGroupsResource(MgmtServiceResourceV3Impl.java:26)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)

avatar
New Contributor

I was able to spend more time looking into logs and figured that this is due to services not able to start. CM instance ran out of memory and went to unstable state. 

 

Increasing the memory resolved this issue. 

avatar
Community Manager

Congratulations on solving the issue and thank you for sharing the solution in case it can assist others. 🙂


Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.