Created on 10-23-2015 11:51 AM - edited 09-16-2022 08:39 AM
When building a new Cluster using Cloudera Director 1.5.1 it fails with the error: "API call to Cloudera Manager failed."
I've experienced the problem 3 times in a row. First when creating Manager and Cluster at the same time, then 2 times when the Manager is already up and running.
I haven't got any other error before this problem, which I'm guessing is one of the latest steps to have a working cluster.
Stacktrace is below.
ERROR [pipeline-thread-80] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=ApiRootResource.getCurrentVersion, Args=null at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:79) ~[launchpad-cloudera-manager-api-ext-1.5.1.jar!/:na] at com.sun.proxy.$Proxy170.getCurrentVersion(Unknown Source) ~[na:na] at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall.run(BootstrapClouderaManagerAgent.java:145) ~[launchpad-bootstrap-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall.run(BootstrapClouderaManagerAgent.java:110) ~[launchpad-bootstrap-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.job.Job3$$FastClassBySpringCGLIB$$54178503.invoke(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.1] at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:55) ~[launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0] at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:51) ~[launchpad-pipeline-1.5.1.jar!/:1.5.1] at sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_65] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_65] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.5.RELEASE.jar!/:4.1.5.RELEASE] at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall$$EnhancerBySpringCGLIB$$a9863238.runUnchecked(<generated>) ~[spring-core-4.1.5.RELEASE.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:165) [launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:136) [launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na] at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na] at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:98) [launchpad-pipeline-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:120) [launchpad-pipeline-database-1.5.1.jar!/:1.5.1] at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-1.5.1.jar!/:1.5.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_65] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Created 10-23-2015 12:30 PM
m3.medium is definitely too small to run Cloudera Manager and all the management services.
My recommendation is to try again with m4.large or m4.xlarge as the Cloudera Manager instance. That should take memory related failures out of the picture.
Created 10-23-2015 11:57 AM
What instance type are you using for Cloudera Manager and for the cluster nodes? What services does the cluster run and on how many instances?
One reason why that API call may fail is if Cloudera Manager runs out of memory. Are you able to access Cloudera Manager with a browser after the failure?
Created 10-23-2015 12:03 PM
Hi, instance type being used for the Manager is m3.medium and for the nodes I'm using m3.xlarge for all but Gateway, which also uses m3.medium.
I've picked the Hadoop with Impala option with no customization. I've noticed CM is not accesible, so probably it haven't started. I'll take a look on that.
Which instance type would you recommend for CM instead?
Created 10-23-2015 12:30 PM
m3.medium is definitely too small to run Cloudera Manager and all the management services.
My recommendation is to try again with m4.large or m4.xlarge as the Cloudera Manager instance. That should take memory related failures out of the picture.
Created 10-23-2015 01:06 PM
Awesome. Thank you.