While running the 'cloudera-director bootstrap' command for the cloudera director client we see the following error in the logs while running the 'Installing cloudera-manager-daemons package' step.  This exact configuration was used to start a cluster the previous day so this is probably something with the environment but I can't find any indications of what may be the cause.   cloudera-director bootstrap cluster.conf Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log Plugins will be loaded from /var/lib/cloudera-director-client/plugins Cloudera Director 1.5.2 initializing ... Installing Cloudera Manager ... * Starting ...... done * Requesting an instance for Cloudera Manager .............................. done * Inspecting capabilities of 10.172.4.89 .......... done * Installing screen package (1/1) .......... done * Running custom bootstrap script on 10.172.4.89 ......... done * Waiting for SSH access to 10.172.4.89 on port 22 ....... done * Inspecting capabilities of 10.172.4.89 ............... done * Normalizing 10.172.4.89 ....... done * Installing ntp package (1/4) ...... done * Installing curl package (2/4) ....... done * Installing nscd package (3/4) ....... done * Installing gdisk package (4/4) ......................... done * Resizing instance root partition ............ done * Rebooting 10.172.4.89 ..... done * Waiting for 10.172.4.89 to boot ....... done * Mounting all instance disk drives ............ done * Waiting for new external database servers to start running ........... done * Installing repositories for Cloudera Manager ......... done * Installing cloudera-manager-daemons package (1/2) .......   [2016-01-05 18:46:26] INFO [io-thread-6] - ssh:10.172.4.89: Error: No matching Packages to list [2016-01-05 18:46:26] INFO [io-thread-6] - ssh:10.172.4.89: http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.4/repodata/repomd.xml: [Errno 14] PYCURL ERROR 56 - "Failure when receiving data from the peer" [2016-01-05 18:46:26] INFO [io-thread-6] - ssh:10.172.4.89: Trying other mirror. [2016-01-05 18:46:26] INFO [io-thread-6] - ssh:10.172.4.89: Error: Cannot retrieve repository metadata (repomd.xml) for repository: cloudera-manager. Please verify its path and try again [2016-01-05 18:46:27] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=500, pipeline=022613a2-a9e2-49e1-86cf-b4d35445a285, ... [2016-01-05 18:46:27] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> SshJobFailFastWithOutputLogging/3 [sudo yum -C list installed 'cloudera-manager-daemons' 2>&1 > /dev/null, 10.172.4.89, SshCredentials ... [2016-01-05 18:46:27] INFO [pipeline-thread-1] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection. [2016-01-05 18:46:27] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.172.4.89 was automatically accepted [2016-01-05 18:46:28] INFO [io-thread-6] - ssh:10.172.4.89: Error: No matching Packages to list [2016-01-05 18:46:28] ERROR [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.launchpad.common.ssh.SshException: Script execution failed with code 1. Script: sudo yum -C list installed 'cloudera-manager-daemons' 2>&1 > /dev/null at com.cloudera.launchpad.pipeline.ssh.SshJobFailFastWithOutputLogging.run(SshJobFailFastWithOutputLogging.java:45) ~[launchpad-pipeline-common-1.5.2.jar!/:1.5.2] at com.cloudera.launchpad.pipeline.ssh.SshJobFailFastWithOutputLogging.run(SshJobFailFastWithOutputLogging.java:27) ~[launchpad-pipeline-common-1.5.2.jar!/:1.5.2] at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2] at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:55) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2] at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0] at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:51) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2] at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.6.0_30] at java.lang.reflect.Method.invoke(Method.java:622) ~[na:1.6.0_30] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at com.cloudera.launchpad.pipeline.ssh.SshJobFailFastWithOutputLogging$$EnhancerBySpringCGLIB$$cde3fbd8.runUnchecked(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.2] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:165) [launchpad-pipeline-1.5.2.jar!/:1.5.2] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:136) [launchpad-pipeline-1.5.2.jar!/:1.5.2] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na] at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na] at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:98) ~[launchpad-pipeline-1.5.2.jar!/:1.5.2] at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:120) ~[launchpad-pipeline-database-1.5.2.jar!/:1.5.2] at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) ~[launchpad-common-1.5.2.jar!/:1.5.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_30] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_30] at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_30] at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30] I am able to manually SSH into the server and try to run the cloudera-manager-daemons installation and I get this output.   [ec2-user@i-23b5d6a2.poc-dis11.daws1 ~]$ sudo yum -C list installed 'cloudera-manager-daemons' Loaded plugins: amazon-id, lsbvars, rhui-lb, security Error: No matching Packages to list [ec2-user@i-23b5d6a2.poc-dis11.daws1 ~]$ sudo yum -C install 'cloudera-manager-daemons' Loaded plugins: amazon-id, lsbvars, rhui-lb, security Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package cloudera-manager-daemons.x86_64 0:5.4.9-1.cm549.p0.9.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: cloudera-manager-daemons x86_64 5.4.9-1.cm549.p0.9.el6 cloudera-manager 638 M Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package(s) Total download size: 638 M Installed size: 902 M Is this ok [y/N]: y Downloading Packages: Error Downloading Packages: cloudera-manager-daemons-5.4.9-1.cm549.p0.9.el6.x86_64: Caching enabled but no local cache of /var/cache/yum/x86_64/6Server/cloudera-manager/packages/cloudera-manager-daemons-5.4.9-1.cm549.p0.9.el6.x86_64.rpm from cloudera-manager Since I know of no changes in our files, I am not sure why the yum / rpm installations from cloudera no longer appear to be working.   ... View more

Hello, I am working on a project to automate a development cluster startup on AWS using cloudera director 1.5.1.  For our purposes we would like to be able to bootstrap and terminate this cluster on a daily basis.  I was successfully able to bootstrap the cluster the first time however now I am getting an error when cloudera director is bootstrapping the Cloudera Manager node.  The high level logs are below.   Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log Plugins will be loaded from /var/lib/cloudera-director-client/plugins Cloudera Director 1.5.1 initializing ... Installing Cloudera Manager ... * Starting ...... done * Requesting an instance for Cloudera Manager ............................ done * Inspecting capabilities of 10.172.4.37 .......... done * Installing screen package (1/1) ........ done * Running custom bootstrap script on 10.172.4.37 ....... done * Waiting for SSH access to 10.172.4.37 on port 22 ...... done * Inspecting capabilities of 10.172.4.37 ................ done * Normalizing 10.172.4.37 ....... done * Installing ntp package (1/4) ...... done * Installing curl package (2/4) ...... done * Installing nscd package (3/4) ...... done * Installing gdisk package (4/4) ......................... done * Resizing instance root partition ............ done * Rebooting 10.172.4.37 .... done * Waiting for 10.172.4.37 to boot ...... done * Mounting all instance disk drives ........... done * Waiting for new external database servers to start running .......... done * Installing repositories for Cloudera Manager ......... done * Installing cloudera-manager-daemons package (1/2) ...... done * Installing cloudera-manager-server package (2/2) ...... done * Configuring external POSTGRESQL database for Cloudera Manager ........ done * Starting Cloudera Manager server ... done * Waiting for Cloudera Manager server to start ...... done * Setting Cloudera Manager License ... done * ClouderaManagerException{message="API call to Cloudera Manager failed. Method=ClouderaManagerResourceV6.beginTrial, Args=null",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"} ... The issue is when cloudera director calls the begin trial API on Cloudera manager and the stack trace is included below.  I don't think anything has changed since the last time this ran successfully.   [2015-11-05 10:41:59] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=ClouderaManagerResourceV6.beginTrial, Args=null",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"} at com.cloudera.launchpad.bootstrap.deployment.LicenseClouderaManager.run(LicenseClouderaManager.java:84) ~[launchpad-bootstrap-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.bootstrap.deployment.LicenseClouderaManager.run(LicenseClouderaManager.java:34) ~[launchpad-bootstrap-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.1] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:55) ~[launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0] at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:51) ~[launchpad-pipeline-1.5.1.jar!/:1.5.1] at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.6.0_30] at java.lang.reflect.Method.invoke(Method.java:622) ~[na:1.6.0_30] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at com.cloudera.launchpad.bootstrap.deployment.LicenseClouderaManager$$EnhancerBySpringCGLIB$$4c61360c.runUnchecked(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:165) ~[launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:136) ~[launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na] at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na] at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:98) ~[launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:120) ~[launchpad-pipeline-database-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) ~[launchpad-common-1.5.1.jar!/:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_30] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_30] at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_30] at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30] Caused by: com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=ClouderaManagerResourceV6.beginTrial, Args=null at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:79) ~[launchpad-cloudera-manager-api-ext-1.5.1.jar!/:na] at com.sun.proxy.$Proxy146.beginTrial(Unknown Source) ~[na:na] at com.cloudera.launchpad.bootstrap.deployment.LicenseClouderaManager.run(LicenseClouderaManager.java:77) ~[launchpad-bootstrap-1.5.1.jar!/:1.5.1] ... 34 common frames omitted Any help would be appreciated, Thanks ... View more