Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Exception when adding Cloudera Manager via the Cloudera Director web UI

avatar
Explorer

Hi everyone,

 

I successfully set a Virtual Private Cloud on AWS and started an instance using the Cloudera Director AMI (ID ami-2957655e, created on October 16).

 

I logged into its Web panel, added an environment, and then attempted to add Cloudera Manager. I did not attempt to go as far as adding a cluster too, because I keep getting a bootstrap_failed error. This has happened a couple of times already, with the same error.

 

(OS: the tmplate for Manager and cluster instances uses an Amazon provisioned AMI running CentOS 6.5. The template uses t2.micro instances. Very small, I know, but this is excludsively to learn the workflow and how to use the interface. Once I see everything running smoothly, I will switch to larger sizes. What follows doesn't seem to be a memory issue anyway.)

 

I checked the /var/log/cloudera-director-server/application.log and I can see that the Director can SSH into the other instance, but then it throws this exception, something to do with Google's Guava library:

 

[2015-10-24 00:17:19] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Pipeline 1efc9b36-50f9-4eca-874d-6a736779de59 suspended due to failure
java.lang.IllegalStateException: Optional.get() cannot be called on an absent value
        at com.google.common.base.Absent.get(Absent.java:42) ~[guava-15.0.jar!/:na]
        at com.cloudera.launchpad.inspector.LogInstalledPackagesAndRepositories.run(LogInstalledPackagesAndRepositories.java:37) ~[launchpad-inspector-1.5.0.jar!/:1.5.0]
        at com.cloudera.launchpad.inspector.LogInstalledPackagesAndRepositories.run(LogInstalledPackagesAndRepositories.java:23) ~[launchpad-inspector-1.5.0.jar!/:1.5.0]
        at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-1.5.0.jar!/:1.5.0]
        at com.cloudera.launchpad.pipeline.job.Job2$$FastClassBySpringCGLIB$$54178502.invoke(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.0]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:55) ~[launchpad-pipeline-1.5.0.jar!/:1.5.0]
        at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:51) ~[launchpad-pipeline-1.5.0.jar!/:1.5.0]
        at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_67]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE]
        at com.cloudera.launchpad.inspector.LogInstalledPackagesAndRepositories$$EnhancerBySpringCGLIB$$5826f59a.runUnchecked(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.0]
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:165) ~[launchpad-pipeline-1.5.0.jar!/:1.5.0]
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:136) ~[launchpad-pipeline-1.5.0.jar!/:1.5.0]
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
        at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:98) ~[launchpad-pipeline-1.5.0.jar!/:1.5.0]
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:120) ~[launchpad-pipeline-database-1.5.0.jar!/:1.5.0]
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-1.5.0.jar!/:1.5.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_67]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
[2015-10-24 00:17:19] INFO  [pipeline-thread-1] - c.c.l.p.s.PipelineRepositoryService: Pipeline '1efc9b36-50f9-4eca-874d-6a736779de59': RUNNING -> SUSPENDED
[2015-10-24 00:17:19] INFO  [pipeline-thread-1] - c.c.l.d.DeploymentRepositoryService: Deployment 'Geomesa shepherd': BOOTSTRAP_FAILED -> BOOTSTRAP_FAILED

Any help is much appreciated.

 

If it is difficult to gauge what the problem is, maybe someone can suggest a tried and tested AMI that I could use for the template.

 

Thanks!

2 ACCEPTED SOLUTIONS

avatar
Master Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Master Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
12 REPLIES 12

avatar
Master Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

Thank you, Andrei. That was so fast!

 

I will do as you suggest, and then come back to mark the answer as correct if everything works.

avatar
Explorer

I thought that it might be useful to also see some of the output before the exception:

 

[2015-10-24 00:17:16] INFO  [pipeline-thread-1] - c.cloudera.launchpad.sshj.SshJClient: Attempting SSH connection.
[2015-10-24 00:17:16] WARN  [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 10.0.0.69 was automatically accepted
[2015-10-24 00:17:16] INFO  [io-thread-1] - ssh:10.0.0.69: CentOS release 6.5 (Final)
[2015-10-24 00:17:16] INFO  [io-thread-1] - ssh:10.0.0.69: Kernel \r on an \m
[2015-10-24 00:17:16] INFO  [io-thread-2] - ssh:10.0.0.69: bash: sudo: command not found
[2015-10-24 00:17:16] INFO  [io-thread-1] - ssh:10.0.0.69: bash: apt-get: command not found
[2015-10-24 00:17:16] INFO  [io-thread-1] - ssh:10.0.0.69: bash: zypper: command not found
[2015-10-24 00:17:17] INFO  [io-thread-1] - ssh:10.0.0.69: bash: java: command not found
[2015-10-24 00:17:17] INFO  [io-thread-1] - ssh:10.0.0.69: Python 2.6.6
[2015-10-24 00:17:17] INFO  [io-thread-2] - ssh:10.0.0.69: bash: sudo: command not found
[2015-10-24 00:17:17] WARN  [pipeline-thread-1] - c.c.l.i.OperatingSystemCapabilitiesInspector: 'sudo' command not found
[2015-10-24 00:17:17] INFO  [io-thread-2] - ssh:10.0.0.69: SELinux status:                 disabled
[2015-10-24 00:17:17] INFO  [io-thread-2] - ssh:10.0.0.69: bash: sudo: command not found
[2015-10-24 00:17:17] WARN  [pipeline-thread-1] - c.c.l.i.OperatingSystemCapabilitiesInspector: 'iptables-save' command not found
[2015-10-24 00:17:17] INFO  [io-thread-2] - ssh:10.0.0.69: ip-10-0-0-69.eu-west-1.compute.internal
[2015-10-24 00:17:17] INFO  [io-thread-2] - ssh:10.0.0.69: bash: nslookup: command not found
[2015-10-24 00:17:17] WARN  [pipeline-thread-1] - c.c.l.i.OperatingSystemCapabilitiesInspector: 'nslookup' command not found
[2015-10-24 00:17:19] INFO  [io-thread-2] - ssh:10.0.0.69: ip-10-0-0-69.eu-west-1.compute.internal
[2015-10-24 00:17:19] INFO  [io-thread-2] - ssh:10.0.0.69: 10.0.0.69
[2015-10-24 00:17:19] DEBUG [pipeline-thread-1] - c.c.l.i.OperatingSystemCapabilitiesInspector: FQDN reported is ip-10-0-0-69.eu-west-1.compute.internal
[2015-10-24 00:17:19] INFO  [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=103, pipeline=1efc9b36-50f9-4eca-874d-6a736779de59,  ...
[2015-10-24 00:17:19] INFO  [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> LogInstalledPackagesAndRepositories/2 [PluggableComputeInstance{ipAddress=Optional.of(10.0.0.69), delegate=null} Instance{virtualInstance= ...
[2015-10-24 00:17:19] ERROR [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
java.lang.IllegalStateException: Optional.get() cannot be called on an absent value

Cheers

avatar
Master Collaborator

That failing DNS could indicate either a problem with CentOS 6.5 AMI you are using or with the VPC DNS  settings.

 

Can you share the CentOS AMI ID? For VPC are you using AmazonProvidedDNS or a custom server? 

avatar
Explorer

Here it is:

 

ID: ami-944a0be3

 

AMI Name: Amazon_CentOS_6-5-x86-64_1.0rev15_2015-Jun-29_12.39.00
 
I'm using AmazonProvidedDNS for VPC.

avatar
Master Collaborator
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

Thanks!

 

That's what I just did. I'm waiting now for the bootstraping of the Manager, on exactly this AMI, in an t3.xlarge instance.

 

Just as a side note. Earlier, before I started writing on this forum, I started a separate installation of Cloudera Manager, via CLI, using a micro instance and that Amazon version of CentOS - just like in the earlier setup. It has finished successfuly now; I only need to give it  the IPs of workers.

avatar
Explorer

The official CentOS 6.5 marketplace AMI with ID: ami-42718735, on a larger instance, and the Cloudera Manager was added successfully.

 

Thank you for your support!

avatar
Master Collaborator

That's great! Can you share more about your use case? What services are you planning to run part of the cluster? Is this going to be a long running cluster?