question Re: com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation in Archives of Support Questions (Read Only)

com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation fai

nayab — Fri, 16 Sep 2022 12:07:10 GMT

Trying to create CDH cluster on AWS but have been running into several issues. The latest is below. I searched this site for other possible solutions like changing the instance type, changing node count etc but without any success.

Appreciate any help on this.

AMI - ami-dbc96ab0

Version: Cloudera Director 2.5.0

[2017-08-18 20:01:54.916 -0400] ERROR [p-782ec459d24c-BootstrapClouderaManagerAgent] b5ec8728-cae0-4fb2-8c37-d3188b02de61 POST /api/v9/environments/gmp-cloudera-aws/deployments/gmp-cloudera-manager1/clusters com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.b.c.BootstrapClouderaManagerAgent: Command GlobalHostInstall with ID 278 failed after 5 tries. Details: ApiCommand{id=278, name=GlobalHostInstall, startTime=Fri Aug 18 20:01:39 EDT 2017, endTime=Fri Aug 18 20:01:54 EDT 2017, active=false, success=false, resultMessage=Command completed with 0/1 successful subcommands, serviceRef=null, roleRef=null, hostRef=null, parent=null}
[2017-08-18 20:01:54.925 -0400] ERROR [p-782ec459d24c-BootstrapClouderaManagerAgent] b5ec8728-cae0-4fb2-8c37-d3188b02de61 POST /api/v9/environments/gmp-cloudera-aws/deployments/gmp-cloudera-manager1/clusters com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation failed on instance '172.16.50.118' after 5 tries.
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:350)
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:294)
at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34)
at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.GeneratedMethodAccessor220.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure$$EnhancerBySpringCGLIB$$9c6fc38b.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:197)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:168)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:133)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:164)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2017-08-18 20:01:54.926 -0400] ERROR [sockmon] - - - - - net.schmizz.concurrent.Promise: <<copyDone>> woke to: net.schmizz.sshj.transport.TransportException: [BY_APPLICATION] Disconnected
[2017-08-18 20:01:54.930 -0400] ERROR [p-782ec459d24c-BootstrapClouderaManagerAgent] b5ec8728-cae0-4fb2-8c37-d3188b02de61 POST /api/v9/environments/gmp-cloudera-aws/deployments/gmp-cloudera-manager1/clusters com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error ErrorInfo{code=CM_AGENT_INSTALLATION_FAIL, properties={instanceIpAddress=172.16.50.118, retryCount=5}, causes=[]} in job com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation failed on instance '172.16.50.118' after 5 tries.

Re: com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation

nayab — Sat, 19 Aug 2017 18:04:43 GMT

It was resolved.

I made security group to allow all traffick inbound and outbound and CDH version to 5.12 instead of just 5.

Not sure which change fixed it.

Re: com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation

jadair — Wed, 23 Aug 2017 21:10:42 GMT

I'm glad your problem is resolved. Thank you for posting the details so other forum members can benefit from your experience.

Re: com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation

Dipa — Thu, 12 Jul 2018 11:55:14 GMT

When I add a deployment for the CDH using Cloudera Director 2.8, the process fails at the 18th step. Can someone please help.

Below is the logs from the application.log

[2018-07-12 10:44:35.977 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:37.478 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:38.979 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:40.480 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:40.982 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Failed to find an accessible endpoint from the list of endpoints [10.3.0.4:22]
[2018-07-12 10:44:40.986 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1011, pipeline=109ef4c4-17e6-486a-a2b1-da8574c7d785/ ...
[2018-07-12 10:44:41.041 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.pipeline.util.PipelineRunner: >> FailIfUnequal/3 [false, true, ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseSer ...
[2018-07-12 10:44:41.045 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: null
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24)
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17)
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32)
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2018-07-12 10:44:41.055 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error: JobError{jobClassName=com.cloudera.launchpad.pipeline.FailIfUnequal, jobContext=JobContext{callCountAtThisStackLevel=0, pipelineHandle='109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab', callStack=CallStack{items=[Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances.WaitForSshToSucceed', callCount=1}, Item{className='com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime', callCount=1}], size=7, parent=Optional.of(CallStack{items=[Item{className='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob', callCount=1}, Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances', callCount=1}, Item{className='com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs', callCount=0}], size=5, parent=Optional.absent()})}, stackLevel=2}, errorInfo=ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='10.3.0.4', hostAddress=Optional.of(/10.3.0.4)}, port=Optional.absent(), url=Optional.absent()}]}, causes=[]}}
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: null
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24)
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17)
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32)
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2018-07-12 10:44:41.057 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.DatabasePipelineRunner: Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab' failed
at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554
at com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime:1
at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs:0

[2018-07-12 10:44:41.071 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.s.PipelineRepositoryService: Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab': RUNNING -> ERROR
[2018-07-12 10:44:41.293 +0000] INFO [p-da8574c7d785-DefaultBootstrapDeploymentJob] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults - c.c.l.p.u.UnboundedParallelForEach: Error while running parallel job.
java.util.concurrent.ExecutionException: PipelineExecutionException{causeClass=class com.cloudera.launchpad.pipeline.UnrecoverablePipelineError, causeStackTrace='com.cloudera.launchpad.pipeline.UnrecoverablePipelineError
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24)
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17)
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32)
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
', pipelineStackTrace='Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab' failed
at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554
at com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime:1
at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs:0
'}
at com.cloudera.launchpad.pipeline.DatabaseValue.getValueOrPropagateException(DatabaseValue.java:128)
at com.cloudera.launchpad.pipeline.DatabaseValue.get(DatabaseValue.java:111)
at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults.runUnchecked(UnboundedParallelForEach.java:157)
at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults$$FastClassBySpringCGLIB$$5477796e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults$$EnhancerBySpringCGLIB$$87c13ed2.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloudera.launchpad.pipeline.PipelineExecutionException: null
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:224)
... 7 common frames omitted

Re: com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation

Dipa — Thu, 12 Jul 2018 12:02:47 GMT

I get the following error when adding a deployment for CDH using Cloudera Director. The process fails at the 18th step. Can someone please help.

[2018-07-12 10:44:29.972 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:31.474 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:32.975 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:34.476 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:35.977 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:37.478 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:38.979 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:40.480 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22
[2018-07-12 10:44:40.982 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Failed to find an accessible endpoint from the list of endpoints [10.3.0.4:22]
[2018-07-12 10:44:40.986 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1011, pipeline=109ef4c4-17e6-486a-a2b1-da8574c7d785/ ...
[2018-07-12 10:44:41.041 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.pipeline.util.PipelineRunner: >> FailIfUnequal/3 [false, true, ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseSer ...
[2018-07-12 10:44:41.045 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: null
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24)
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17)
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32)
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2018-07-12 10:44:41.055 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error: JobError{jobClassName=com.cloudera.launchpad.pipeline.FailIfUnequal, jobContext=JobContext{callCountAtThisStackLevel=0, pipelineHandle='109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab', callStack=CallStack{items=[Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances.WaitForSshToSucceed', callCount=1}, Item{className='com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime', callCount=1}], size=7, parent=Optional.of(CallStack{items=[Item{className='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob', callCount=1}, Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances', callCount=1}, Item{className='com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs', callCount=0}], size=5, parent=Optional.absent()})}, stackLevel=2}, errorInfo=ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='10.3.0.4', hostAddress=Optional.of(/10.3.0.4)}, port=Optional.absent(), url=Optional.absent()}]}, causes=[]}}
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: null
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24)
at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17)
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32)
at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2018-07-12 10:44:41.057 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.DatabasePipelineRunner: Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab' failed
at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554
at com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime:1
at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs:0