Reply
New Contributor
Posts: 5
Registered: ‎07-19-2018

Cluster Bootstrap Fails Google Cloud

I've seen other UnrecoverablePipelineError bootstrapping failure posts, but none of the solutions seem to help me. Followed this https://www.cloudera.com/documentation/director/latest/topics/director_get_started_gcp.html and skipped the SOCKS proxy step.

 

CM 5.14, centos 6, n1-highmem-4, Boot Disk Size: 500GB, Data Disk Count: 0

Core Hadoop

CDH 5.14, same template as CM

1 master, 3 workers, 1 gateway

 

The only errors I see in the logs before are

[2018-07-20 03:36:23.559 +0000] ERROR [port-forwarding-[45540:10.142.0.3:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#0 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2018-07-20 03:36:33.560 +0000] INFO  [p-89cf80421317-DefaultBootstrapDeploymentJob] 8404c507-c96d-49b1-8c16-7c2c6c1db11f POST /api/v12/environments/hackathon2/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.launchpad.sshj.SshJPortForwarder: Starting port forwarding thread
[2018-07-20 03:36:33.571 +0000] ERROR [port-forwarding-[35709:10.142.0.3:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#1 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2018-07-20 03:36:43.572 +0000] INFO  [p-89cf80421317-DefaultBootstrapDeploymentJob] 8404c507-c96d-49b1-8c16-7c2c6c1db11f POST /api/v12/environments/hackathon2/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.launchpad.sshj.SshJPortForwarder: Starting port forwarding thread
[2018-07-20 03:36:43.575 +0000] ERROR [port-forwarding-[36225:10.142.0.3:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#2 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2018-07-20 03:36:53.575 +0000] INFO  [p-89cf80421317-DefaultBootstrapDeploymentJob] 8404c507-c96d-49b1-8c16-7c2c6c1db11f POST /api/v12/environments/hackathon2/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.launchpad.sshj.SshJPortForwarder: Starting port forwarding thread
[2018-07-20 03:36:53.585 +0000] ERROR [port-forwarding-[41276:10.142.0.3:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#3 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2018-07-20 03:36:59.134 +0000] INFO  [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing all deployments for usage reporting
[2018-07-20 03:36:59.136 +0000] INFO  [task-thread-2] - - - - - c.c.l.m.r.DeploymentsReporter: Enqueueing 0 deployments for usage reporting
[2018-07-20 03:37:03.586 +0000] INFO  [p-89cf80421317-DefaultBootstrapDeploymentJob] 8404c507-c96d-49b1-8c16-7c2c6c1db11f POST /api/v12/environments/hackathon2/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.launchpad.sshj.SshJPortForwarder: Starting port forwarding thread
[2018-07-20 03:37:03.589 +0000] ERROR [port-forwarding-[40858:10.142.0.3:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#4 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2018-07-20 03:37:13.590 +0000] INFO  [p-89cf80421317-DefaultBootstrapDeploymentJob] 8404c507-c96d-49b1-8c16-7c2c6c1db11f POST /api/v12/environments/hackathon2/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.launchpad.sshj.SshJPortForwarder: Starting port forwarding thread
[2018-07-20 03:37:13.591 +0000] ERROR [port-forwarding-[42787:10.142.0.3:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#5 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused
[2018-07-20 03:37:23.592 +0000] INFO  [p-89cf80421317-DefaultBootstrapDeploymentJob] 8404c507-c96d-49b1-8c16-7c2c6c1db11f POST /api/v12/environments/hackathon2/deployments com.cloudera.launchpad.bootstrap.WaitForSshTunnelToServerOnPort - c.c.launchpad.sshj.SshJPortForwarder: Starting port forwarding thread
[2018-07-20 03:37:23.595 +0000] ERROR [port-forwarding-[46296:10.142.0.3:7180]] - - - - - net.schmizz.concurrent.Promise: <<chan#6 / open>> woke to: Opening `direct-tcpip` channel failed: Connection refused

 Then the UnrecoverablePipelineError

[2018-07-20 03:42:18.593 +0000] WARN  [p-d9eea61eef1b-AllocateInstances] 42691f61-43ae-4776-ae7a-7c31ea7511f8 POST /api/v12/environments/hackathon2/deployments/hackathon_deployment/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.bootstrap.AllocateInstances: Error while attempting to allocate instances for group masters. Attempting to continue.
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Problem allocating instances.
	at com.cloudera.launchpad.pluggable.common.ExceptionConditions.propagateWithSuppressedDetails(ExceptionConditions.java:126)
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:616)
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocateInstancesForTemplate(PluggableComputeProvider.java:545)
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:520)
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:331)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:221)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:196)
	at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
	at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
	at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$61e27a3d.runUnchecked(<generated>)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
	at com.github.rholder.retry.Retryer.call(Retryer.java:160)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: com.cloudera.launchpad.pluggable.common.ExceptionConditions$DetailHolderException: Exception details:
  key: null
    PluginExceptionCondition{type=ERROR, exceptionInfo={message=403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Quota 'CPUS' exceeded. Limit: 8.0 in region us-east1.",
    "reason" : "quotaExceeded"
  } ],
  "message" : "Quota 'CPUS' exceeded. Limit: 8.0 in region us-east1."
}}}

Caused by: com.cloudera.director.spi.v2.model.exception.UnrecoverableProviderException: Problem allocating instances.
	at com.cloudera.director.spi.v2.adapters.v1.ExceptionsAdapter.fromV1(ExceptionsAdapter.java:84)
	at com.cloudera.director.spi.v2.adapters.v1.ExceptionsAdapter.fromV1(ExceptionsAdapter.java:51)
	at com.cloudera.director.spi.v2.adapters.v1.ComputeProviderAdapter$1.allocate(ComputeProviderAdapter.java:125)
	at com.cloudera.director.spi.v2.adapters.v1.ComputeProviderAdapter$1.allocate(ComputeProviderAdapter.java:55)
	at com.cloudera.launchpad.pluggable.compute.PluggableComputeProvider.allocate(PluggableComputeProvider.java:614)
	... 36 common frames omitted
[2018-07-20 03:42:18.594 +0000] ERROR [p-d9eea61eef1b-AllocateInstances] 42691f61-43ae-4776-ae7a-7c31ea7511f8 POST /api/v12/environments/hackathon2/deployments/hackathon_deployment/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available after allocation
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:265)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:196)
	at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
	at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
	at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$61e27a3d.runUnchecked(<generated>)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
	at com.github.rholder.retry.Retryer.call(Retryer.java:160)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2018-07-20 03:42:18.594 +0000] ERROR [p-d9eea61eef1b-AllocateInstances] 42691f61-43ae-4776-ae7a-7c31ea7511f8 POST /api/v12/environments/hackathon2/deployments/hackathon_deployment/clusters com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error: JobError{jobClassName=com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun, jobContext=JobContext{callCountAtThisStackLevel=0, pipelineHandle='9e653d29-23d5-4a4e-9916-9980398041d2/child-00000-1dec22a8-e73a-4d21-a290-d9eea61eef1b', callStack=CallStack{items=[Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances', callCount=0}], size=5, parent=Optional.of(CallStack{items=[Item{className='com.cloudera.launchpad.api.jobs.DefaultBootstrapClusterJob', callCount=1}, Item{className='com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs', callCount=0}], size=4, parent=Optional.absent()})}, stackLevel=1}, errorInfo=ErrorInfo{code=INSTANCE_ALLOCATION_FAILURE, properties={minRequiredCount=1, numberOfInstancesAllocated=0, virtualInstanceGroup=masters}, causes=[ErrorInfo{code=PROVIDER_ERROR, properties={details={"message":"403 Forbidden\n{\n  \"code\" : 403,\n  \"errors\" : [ {\n    \"domain\" : \"usageLimits\",\n    \"message\" : \"Quota 'CPUS' exceeded. Limit: 8.0 in region us-east1.\",\n    \"reason\" : \"quotaExceeded\"\n  } ],\n  \"message\" : \"Quota 'CPUS' exceeded. Limit: 8.0 in region us-east1.\"\n}"}}, causes=[]}]}}
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Insufficient number of instances available after allocation
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:265)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun.run(AllocateInstances.java:196)
	at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31)
	at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
	at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
	at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
	at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
	at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at com.cloudera.launchpad.bootstrap.AllocateInstances$AllocateAndWaitForInstancesToRun$$EnhancerBySpringCGLIB$$61e27a3d.runUnchecked(<generated>)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
	at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
	at com.github.rholder.retry.Retryer.call(Retryer.java:160)
	at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
	at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
	at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Any thoughts?

Cloudera Employee
Posts: 74
Registered: ‎10-28-2014

Re: Cluster Bootstrap Fails Google Cloud

ppiper,

 

The port forwarding errors can generally be ignored. Cloudera Director is polling the ssh port on the instance to wait until the instance starts responding.

 

The failure is likely indicated by this in the second set of logs you included

    PluginExceptionCondition{type=ERROR, exceptionInfo={message=403 Forbidden
{
  "code" : 403,
  "errors" : [ {
    "domain" : "usageLimits",
    "message" : "Quota 'CPUS' exceeded. Limit: 8.0 in region us-east1.",
    "reason" : "quotaExceeded"
  } ],
  "message" : "Quota 'CPUS' exceeded. Limit: 8.0 in region us-east1."
}}}

It looks like you exceeded your quota in Google Cloud.

 

David

New Contributor
Posts: 5
Registered: ‎07-19-2018

Re: Cluster Bootstrap Fails Google Cloud

Ahh, now I feel foolish. This was supposed to be a POC on my trial account before migrating everything over to our corporate account. I'll update this post to confirm as soon as that happens. Thank you.

New Contributor
Posts: 5
Registered: ‎07-19-2018

Re: Cluster Bootstrap Fails Google Cloud

Bumped everything down to n1-standard-1, and the cluster bootsrapping still failed. Guessing I'm going to need to wait til I get access to a non-trail account to use n1-highmem-4 for my cdh nodes

 

[2018-07-20 22:18:01.230 +0000] ERROR [p-5ceb2bf2268c-DefaultBootstrapClusterJob] b6e48808-b9bc-41d0-817b-1f94bcc75f42 POST /api/v12/environments/hackathon2/deployments/hackathon_deployment/clusters com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
javax.ws.rs.ProcessingException: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/55: Connection refused (Connection refused)
        at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:596)
        at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:578)
        at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:748)
        at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:231)
        at com.sun.proxy.$Proxy248.readCommand(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor373.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:117)
        at com.sun.proxy.$Proxy248.readCommand(Unknown Source)
        at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.pollWhileActive(UnboundedWaitForApiCommand.java:118)
        at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:55)
        at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:35)
        at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33)
        at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
        at sun.reflect.GeneratedMethodAccessor211.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
        at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$d3f5bb74.runUnchecked(<generated>)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/55: Connection refused (Connection refused)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1377)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1361)
        at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
        at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651)
        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
        at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:666)
        at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:746)
        ... 41 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
        at sun.net.www.http.HttpClient.New(HttpClient.java:339)
        at sun.net.www.http.HttpClient.New(HttpClient.java:357)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1199)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:332)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1578)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1607)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1551)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348)
        ... 47 common frames omitted
New Contributor
Posts: 5
Registered: ‎07-19-2018

Re: Cluster Bootstrap Fails Google Cloud

This issue was resolved after migrating to a non-trial account and using highmem-4 for the manager and cdh nodes template.

Cloudera Employee
Posts: 62
Registered: ‎10-28-2014

Re: Cluster Bootstrap Fails Google Cloud

Glad you got past this problem!

Announcements