Reply
New Contributor
Posts: 2
Registered: ‎08-09-2017

Director Fails to bootstrap CDH 5.10.1 on Azure

[ Edited ]

Hello,

I'm trying to deploy CDH 5.10.1 using director 2.3 the bootstrap is failing to install cloudera manager agents on the instances. Any pointers on how to fix this issue?

 

ERROR [pipeline-thread-43] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation failed on instance '10.32.6.26' after 5 tries.
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:263)
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:231)
at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34)
at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure$$EnhancerBySpringCGLIB$$140967f1.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:104)
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:128)
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2017-08-09 13:01:23.053 -0500] ERROR [pipeline-thread-43] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: Cloudera Manager agent installation failed on instance '10.32.6.26' after 5 tries.
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:263)
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure.run(BootstrapClouderaManagerAgent.java:231)
at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34)
at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67)
at com.codahale.metrics.Timer.time(Timer.java:101)
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63)
at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$WaitForSuccessOrRetryOnFailure$$EnhancerBySpringCGLIB$$140967f1.runUnchecked(<generated>)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164)
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135)
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
at com.github.rholder.retry.Retryer.call(Retryer.java:160)

 

Thanks

Highlighted
New Contributor
Posts: 2
Registered: ‎08-09-2017

Re: Director Fails to bootstrap Cloudera manager agents

Here is is output from the CLI

 

Cloudera Director 2.3.0 initializing ... Connecting to http://10.32.6.5:7189 Current user roles: [ROLE_READONLY, ROLE_ADMIN] Configuration file passes all validation checks. Creating a new environment... Creating external database servers if configured... Creating a new Cloudera Manager... Creating a new CDH cluster... * Requesting an instance for Cloudera Manager .......... done * Installing screen package (1/1) ..... done * Running custom bootstrap script on [10.32.6.18] .......... done * Inspecting capabilities of 10.32.6.18 ...... done * Normalizing 392f0ac1-4275-4fd3-a9a8-efa75a07288a ... done * Installing ntp package (1/3) .... done * Installing curl package (2/3) .... done * Installing nscd package (3/3) ................... done * Resizing instance root partition ....... done * Rebooting 392f0ac1-4275-4fd3-a9a8-efa75a07288a .... done * Waiting for 392f0ac1-4275-4fd3-a9a8-efa75a07288a to boot ..... done * Mounting all instance disk drives ...... done * Waiting for new external database servers to start running .... done * Installing repositories for Cloudera Manager ... done * Installing cloudera-manager-daemons package (1/2) ..... done * Installing cloudera-manager-server package (2/2) ..... done * Installing mysql-connector-java package (1/2) ..... done * Installing mysql-devel package (2/2) .... done * Installing krb5-workstation package (1/2) ..... done * Installing openldap-clients package (2/2) .... done * Configuring external MYSQL database for Cloudera Manager .... done * Starting Cloudera Manager server ... done * Waiting for Cloudera Manager server to start ... done * Changing admin Credentials for Cloudera Manager ... done * Setting Cloudera Manager License ... done * Applying Cloudera Manager license ... done * Restarting Cloudera Manager ... done * Waiting for Cloudera Manager server to restart ..... done * Configuring Cloudera Manager ... done * Importing Kerberos admin principal credentials into Cloudera Manager ... done * Deploying Cloudera Manager agent .... done * Waiting for Cloudera Manager to deploy agent on 10.32.6.18 ... done * Setting up Cloudera Management Services ......... done * Backing up Cloudera Manager Server configuration .... done * Inspecting capabilities of 10.32.6.18 ...... done * Running Deployment post create scripts ... done * Done ... Cloudera Manager ready. * Waiting for Cloudera Manager installation to complete ......... done * Installing Cloudera Manager agents on all instances in parallel (20 at a time) ......................................................................... done * Starting ... done * Collecting diagnostic data ................................................................................................................................................................................................................................................................................ done * Parallel execution of jobs failed. Total: 8 Canceled: 0 Failed: 8 ...

Announcements