Member since
07-12-2018
2
Posts
0
Kudos Received
0
Solutions
07-12-2018
05:02 AM
I get the following error when adding a deployment for CDH using Cloudera Director. The process fails at the 18th step. Can someone please help. [2018-07-12 10:44:29.972 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:31.474 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:32.975 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:34.476 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:35.977 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:37.478 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:38.979 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:40.480 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:40.982 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Failed to find an accessible endpoint from the list of endpoints [10.3.0.4:22] [2018-07-12 10:44:40.986 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1011, pipeline=109ef4c4-17e6-486a-a2b1-da8574c7d785/ ... [2018-07-12 10:44:41.041 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.pipeline.util.PipelineRunner: >> FailIfUnequal/3 [false, true, ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseSer ... [2018-07-12 10:44:41.045 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: null at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24) at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17) at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2018-07-12 10:44:41.055 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error: JobError{jobClassName=com.cloudera.launchpad.pipeline.FailIfUnequal, jobContext=JobContext{callCountAtThisStackLevel=0, pipelineHandle='109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab', callStack=CallStack{items=[Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances.WaitForSshToSucceed', callCount=1}, Item{className='com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime', callCount=1}], size=7, parent=Optional.of(CallStack{items=[Item{className='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob', callCount=1}, Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances', callCount=1}, Item{className='com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs', callCount=0}], size=5, parent=Optional.absent()})}, stackLevel=2}, errorInfo=ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='10.3.0.4', hostAddress=Optional.of(/10.3.0.4)}, port=Optional.absent(), url=Optional.absent()}]}, causes=[]}} com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: null at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24) at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17) at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2018-07-12 10:44:41.057 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.DatabasePipelineRunner: Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab' failed at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554 at com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime:1 at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs:0 [2018-07-12 10:44:41.071 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.s.PipelineRepositoryService: Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab': RUNNING -> ERROR [2018-07-12 10:44:41.293 +0000] INFO [p-da8574c7d785-DefaultBootstrapDeploymentJob] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults - c.c.l.p.u.UnboundedParallelForEach: Error while running parallel job. java.util.concurrent.ExecutionException: PipelineExecutionException{causeClass=class com.cloudera.launchpad.pipeline.UnrecoverablePipelineError, causeStackTrace='com.cloudera.launchpad.pipeline.UnrecoverablePipelineError at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24) at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17) at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ', pipelineStackTrace='Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab' failed at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554 at com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime:1 at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs:0 '} at com.cloudera.launchpad.pipeline.DatabaseValue.getValueOrPropagateException(DatabaseValue.java:128) at com.cloudera.launchpad.pipeline.DatabaseValue.get(DatabaseValue.java:111) at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults.runUnchecked(UnboundedParallelForEach.java:157) at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults$$FastClassBySpringCGLIB$$5477796e.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults$$EnhancerBySpringCGLIB$$87c13ed2.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.cloudera.launchpad.pipeline.PipelineExecutionException: null at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:224) ... 7 common frames omitted
... View more
07-12-2018
04:55 AM
When I add a deployment for the CDH using Cloudera Director 2.8, the process fails at the 18th step. Can someone please help. Below is the logs from the application.log [2018-07-12 10:44:35.977 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:37.478 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:38.979 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:40.480 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Attempting connection to /10.3.0.4:22 [2018-07-12 10:44:40.982 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.b.WaitForServersUntilTime: Failed to find an accessible endpoint from the list of endpoints [10.3.0.4:22] [2018-07-12 10:44:40.986 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=1011, pipeline=109ef4c4-17e6-486a-a2b1-da8574c7d785/ ... [2018-07-12 10:44:41.041 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.pipeline.util.PipelineRunner: >> FailIfUnequal/3 [false, true, ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseSer ... [2018-07-12 10:44:41.045 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: null at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24) at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17) at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2018-07-12 10:44:41.055 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error: JobError{jobClassName=com.cloudera.launchpad.pipeline.FailIfUnequal, jobContext=JobContext{callCountAtThisStackLevel=0, pipelineHandle='109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab', callStack=CallStack{items=[Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances.WaitForSshToSucceed', callCount=1}, Item{className='com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime', callCount=1}], size=7, parent=Optional.of(CallStack{items=[Item{className='com.cloudera.launchpad.api.jobs.DefaultBootstrapDeploymentJob', callCount=1}, Item{className='com.cloudera.launchpad.bootstrap.AllocateInstances', callCount=1}, Item{className='com.cloudera.launchpad.pipeline.util.ParallelForEachInBatches', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach', callCount=0}, Item{className='com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs', callCount=0}], size=5, parent=Optional.absent()})}, stackLevel=2}, errorInfo=ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='10.3.0.4', hostAddress=Optional.of(/10.3.0.4)}, port=Optional.absent(), url=Optional.absent()}]}, causes=[]}} com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: null at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24) at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17) at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2018-07-12 10:44:41.057 +0000] ERROR [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.DatabasePipelineRunner: Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab' failed at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554 at com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime:1 at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs:0 [2018-07-12 10:44:41.071 +0000] INFO [p-e6e67ab7ccab-WaitForSshToSucceed] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.FailIfUnequal - c.c.l.p.s.PipelineRepositoryService: Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab': RUNNING -> ERROR [2018-07-12 10:44:41.293 +0000] INFO [p-da8574c7d785-DefaultBootstrapDeploymentJob] 4efeab30-027b-4199-865e-4441389805bb POST /api/v12/environments/CEAzureEnv/deployments com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults - c.c.l.p.u.UnboundedParallelForEach: Error while running parallel job. java.util.concurrent.ExecutionException: PipelineExecutionException{causeClass=class com.cloudera.launchpad.pipeline.UnrecoverablePipelineError, causeStackTrace='com.cloudera.launchpad.pipeline.UnrecoverablePipelineError at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:24) at com.cloudera.launchpad.pipeline.FailIfUnequal.run(FailIfUnequal.java:17) at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ', pipelineStackTrace='Pipeline '109ef4c4-17e6-486a-a2b1-da8574c7d785/child-00000-3925b2d7-b3d7-4077-a776-e6e67ab7ccab' failed at com.cloudera.launchpad.pipeline.FailIfUnequal$$EnhancerBySpringCGLIB$$23b55554 at com.cloudera.launchpad.bootstrap.WaitForSshAccessOnInstanceUntilTime:1 at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach.StartChildJobs:0 '} at com.cloudera.launchpad.pipeline.DatabaseValue.getValueOrPropagateException(DatabaseValue.java:128) at com.cloudera.launchpad.pipeline.DatabaseValue.get(DatabaseValue.java:111) at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults.runUnchecked(UnboundedParallelForEach.java:157) at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults$$FastClassBySpringCGLIB$$5477796e.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at com.cloudera.launchpad.pipeline.util.UnboundedParallelForEach$CollectJobResults$$EnhancerBySpringCGLIB$$87c13ed2.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:201) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:172) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:135) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:200) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:140) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.cloudera.launchpad.pipeline.PipelineExecutionException: null at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.doRun(DatabasePipelineRunner.java:224) ... 7 common frames omitted
... View more