Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

Cloudera Director "API call to Cloudera Manager failed". Creating Roles for Service failed

avatar
author-rank ozzielabrat
Rising Star

Created on ‎04-19-2016 05:57 PM - edited ‎09-16-2022 08:40 AM

Hi,

 

Just wondering if anyone has experienced this issue with Director 2….and whether Cloudera Director Support Guru's can advise on it ?

 

Configuration Details:

-----------------------------

 

When using both the Director Server GUI and Director Client "remote-bootstrap" command to try and create a cluster, I'm using:

 

a) The same AWS VPC/Subnet/Security Group/Network ACL's and have inbound rules/outbound rules set to "ALL Traffic".

b) For the Director 2.0 instance:  my own AWS EC2 "AMI" (a "c4.xlarge" instance running CentOS 6.7, fully patched with /root volume set to 50GB).

c) For the Cloudera Manager instance:  my own AWS EC2 "AMI" (a "r3.xlarge" instance running CentOS 6.7, fully patched with /root volume set to 500GB)….that is successfully created by the Director instance above.

d) For the CDH Master/Worker instances:  my own AWS EC2 "AMI" (a "d2.2xlarge" instance running CentOS 6.7, fully patched with /root volume set to 500GB and 6 x 2TB instance store volumes for HDFS)….again, these get created successfully by the Director instance above.

 

 

Issue Occurring:

----------------------

 

1)  When I use the Director Server GUI to create a cluster:

 

All seems to go well, until I get the error shown below in the applications.log file….

 

If I login to the Cloudera Manager GUI, it shows all "Services" (HDFS, HIVE, HUE, OOZIE, YARN, ZOOKEEPER) with red configuration issue beacons next to them.

 

[2016-04-19 12:31:14] INFO  [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> AddServices/5 [Environment{name='DEV_Environment', provider=InstanceProviderConfig{type='aws'}, credentials=SshCre ...

[2016-04-19 12:31:14] INFO  [pipeline-thread-3] - c.c.l.bootstrap.cluster.AddServices: Creating and configuring services [HDFS, HIVE, HUE, OOZIE, SQOOP, YARN, ZOOKEEPER]

[2016-04-19 12:31:14] INFO  [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Creating cluster services

[2016-04-19 12:31:14] INFO  [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Assigning roles to instances

[2016-04-19 12:31:14] INFO  [pipeline-thread-3] - c.c.l.bootstrap.cluster.AddServices: Creating 24 roles for service CD-HDFS-caRUetBM

[2016-04-19 12:31:14] ERROR [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed

com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RolesResource.createRoles",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"}

at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:319) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:98) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

 

 

2)  When I use the Director Server Client CLI (remote-bootstrap command):

 

I have created a new config file on this Director instance (/usr/lib64/cloudera-director/client/client1_dev_cdh_cluster.aws.cluster.conf).

 

As per the section above titled "Configuration Details", the config file references a single AWS VPC Subnet/Security Group (with specific inbound/outbound rules defined in the Director 2.0 User Guide).

 

The config file doesn't use external databases for anything (just the normal H2 database and local PostgreSQL databases for the Cloudera amon/rman/nav/navms/hue/hive metastore etc etc).

 

When I try and create a new cluster using the following "bootstrap-remote" command using the Cloudera Director Client, it fails with the same error as when using Cloudera Director Server GUI.

 

The Director Client "remote-bootstrap" command I am using is shown below:

 

 

[root@]#  cloudera-director bootstrap-remote client1_dev_cdh_cluster.aws.cluster.conf --lp.remote.username=admin --lp.remote.password={obfuscated} --lp.remote.hostAndPort=10.0.1.247:7189

 

Process logs can be found at /root/.cloudera-director/logs/application.log

Plugins will be loaded from /var/lib/cloudera-director-plugins

Cloudera Director 2.0.0 initializing ...

Connecting to http://10.0.1.247:7189

Current user roles: [ROLE_ADMIN, ROLE_READONLY]

Configuration file passes all validation checks.

Creating a new environment...

Creating external database servers if configured...

Creating a new Cloudera Manager...

Creating a new CDH cluster...

* Requesting an instance for Cloudera Manager ...... done

* Installing screen package (1/1) .... done

* Running custom bootstrap script on 10.0.1.240 ..... done

* Inspecting capabilities of 10.0.1.240 ... done

* Normalizing 10.0.1.240 ... done

* Installing ntp package (1/4) ... done

* Installing curl package (2/4) .... done

* Installing nscd package (3/4) .... done

* Installing gdisk package (4/4) ........... done

* Resizing instance root partition .... done

* Mounting all instance disk drives ...... done

* Waiting for new external database servers to start running ... done

* Installing repositories for Cloudera Manager ... done

* Installing cloudera-manager-daemons package (1/2) ... done

* Installing cloudera-manager-server package (2/2) .... done

* Installing cloudera-manager-server-db-2 package (1/1) .... done

* Starting embedded PostgreSQL database ... done

* Starting Cloudera Manager server ... done

* Waiting for Cloudera Manager server to start ...

* Waiting for Cloudera Manager server to start ... done

* Setting Cloudera Manager License ... done

* Enabling Enterprise Trial ... done

* Deploying Cloudera Manager agent ... done

* Waiting for Cloudera Manager to deploy agent on 10.0.1.240 …

* Setting up Cloudera Management Services ......... done

* Inspecting capabilities of 10.0.1.240 ... done

* Done ...

Cloudera Manager ready.

* Preparing instances in parallel (20 at a time) ................................................ done

* Installing Cloudera Manager agents on all instances in parallel (20 at a time)................. done

* Creating cluster: Client1_DEV_CDH_Cluster ......................................................done

* Downloading parcels: CDH-5.5.2-1.cdh5.5.2.p0.4 .................................................done

* Raising rate limits for parcel distribution to 256000KB/s with 5 concurrent uploads.............done

* Distributing parcels: CDH-5.5.2-1.cdh5.5.2.p0.4 ................................................done

* Activating parcels: CDH-5.5.2-1.cdh5.5.2.p0.4 ..................................................done

* Creating cluster services.......................................................................done

* ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RolesResource.createRoles",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"} …

 

 

Extract showing the failure from the /usr/lib64/cloudera-director/server/logs/application.log file is shown below:

 

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> ParallelForEachInBatches/5 [20, class com.cloudera.launchpad.bootstrap.cluster.InstallJdbcDriverPackages, [PluggableComputeInst ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.p.u.ParallelForEachInBatches: Generating batch for job class com.cloudera.launchpad.bootstrap.cluster.InstallJdbcDriverPackages of size 3

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=9105, pipeline=9f8c84b8-8f97-48b6-9def-9cd786eefae1, ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedParallelForEach/4 [class com.cloudera.launchpad.bootstrap.cluster.InstallJdbcDriverPackages, [PluggableComputeInstance ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.p.DatabasePipelineService: Starting pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00000-ba5d7e82-4484-4823-b5dc-e89cdcd5bac4' with root job com.cloudera.launchpad.bootstrap.cluster.InstallJdbcDriverPackages and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.p.DatabasePipelineService: Starting pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00001-9637f2ff-daa9-452d-bcbb-475a6d8e4759' with root job com.cloudera.launchpad.bootstrap.cluster.InstallJdbcDriverPackages and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.pipeline.util.PipelineRunner: >> InstallJdbcDriverPackages/3 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.49), delegate=null} Instance{virtualInstance= ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.pipeline.util.PipelineRunner: << None{}

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.p.DatabasePipelineService: Starting pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00002-b5b6a1fa-9404-4929-bce0-6d0ea967e699' with root job com.cloudera.launchpad.bootstrap.cluster.InstallJdbcDriverPackages and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.p.s.PipelineRepositoryService: Pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00000-ba5d7e82-4484-4823-b5dc-e89cdcd5bac4': RUNNING -> COMPLETED

[2016-04-20 09:06:05] INFO  [pipeline-thread-59] - c.c.l.pipeline.util.PipelineRunner: >> InstallJdbcDriverPackages/3 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.50), delegate=null} Instance{virtualInstance= ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-59] - c.c.l.pipeline.util.PipelineRunner: << None{}

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=9122, pipeline=9f8c84b8-8f97-48b6-9def-9cd786eefae1, ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-59] - c.c.l.p.s.PipelineRepositoryService: Pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00001-9637f2ff-daa9-452d-bcbb-475a6d8e4759': RUNNING -> COMPLETED

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.pipeline.util.PipelineRunner: >> InstallJdbcDriverPackages/3 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.48), delegate=null} Instance{virtualInstance= ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.pipeline.util.PipelineRunner: << None{}

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.p.s.PipelineRepositoryService: Pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00002-b5b6a1fa-9404-4929-bce0-6d0ea967e699': RUNNING -> COMPLETED

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForAllPipelines/3 [9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00000-ba5d7e82-4484-4823-b5dc-e89cdcd5bac4, 9f8c84b8-8f9 ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << None{}

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> WaitForAllValues/3 [null, null, null]

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=9123, pipeline=9f8c84b8-8f97-48b6-9def-9cd786eefae1, ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> CreateList/1 [[]]

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=9124, pipeline=9f8c84b8-8f97-48b6-9def-9cd786eefae1, ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> ParallelForEachInBatches/5 [20, class com.cloudera.launchpad.bootstrap.cluster.InstallKerberosPackages, [PluggableComputeInstan ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.p.u.ParallelForEachInBatches: Generating batch for job class com.cloudera.launchpad.bootstrap.cluster.InstallKerberosPackages of size 3

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=9130, pipeline=9f8c84b8-8f97-48b6-9def-9cd786eefae1, ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedParallelForEach/4 [class com.cloudera.launchpad.bootstrap.cluster.InstallKerberosPackages, [PluggableComputeInstance{i ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.p.DatabasePipelineService: Starting pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00000-781b56ba-9a23-48eb-83fd-edaef2e1b797' with root job com.cloudera.launchpad.bootstrap.cluster.InstallKerberosPackages and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.p.DatabasePipelineService: Starting pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00001-85c97693-7df7-48da-80fa-cd5a75dce057' with root job com.cloudera.launchpad.bootstrap.cluster.InstallKerberosPackages and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.pipeline.util.PipelineRunner: >> InstallKerberosPackages/3 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.49), delegate=null} Instance{virtualInstance= ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.pipeline.util.PipelineRunner: << None{}

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.p.DatabasePipelineService: Starting pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00002-f7cbec2b-6f4c-4da7-8e4e-77a32bdb521c' with root job com.cloudera.launchpad.bootstrap.cluster.InstallKerberosPackages and listener com.cloudera.launchpad.pipeline.listener.NoopPipelineStageListener

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.p.s.PipelineRepositoryService: Pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00000-781b56ba-9a23-48eb-83fd-edaef2e1b797': RUNNING -> COMPLETED

[2016-04-20 09:06:05] INFO  [pipeline-thread-59] - c.c.l.pipeline.util.PipelineRunner: >> InstallKerberosPackages/3 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.50), delegate=null} Instance{virtualInstance= ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-59] - c.c.l.pipeline.util.PipelineRunner: << None{}

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=9147, pipeline=9f8c84b8-8f97-48b6-9def-9cd786eefae1, ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-59] - c.c.l.p.s.PipelineRepositoryService: Pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00001-85c97693-7df7-48da-80fa-cd5a75dce057': RUNNING -> COMPLETED

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.pipeline.util.PipelineRunner: >> InstallKerberosPackages/3 [PluggableComputeInstance{ipAddress=Optional.of(10.0.1.48), delegate=null} Instance{virtualInstance= ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.pipeline.util.PipelineRunner: << None{}

[2016-04-20 09:06:05] INFO  [pipeline-thread-58] - c.c.l.p.s.PipelineRepositoryService: Pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00002-f7cbec2b-6f4c-4da7-8e4e-77a32bdb521c': RUNNING -> COMPLETED

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForAllPipelines/3 [9f8c84b8-8f97-48b6-9def-9cd786eefae1/child-00000-781b56ba-9a23-48eb-83fd-edaef2e1b797, 9f8c84b8-8f9 ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << None{}

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> WaitForAllValues/3 [null, null, null]

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=9148, pipeline=9f8c84b8-8f97-48b6-9def-9cd786eefae1, ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> CreateList/1 [[]]

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=9149, pipeline=9f8c84b8-8f97-48b6-9def-9cd786eefae1, ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: >> AddServices/5 [Environment{name='Client1_DEV_CDH_Cluster Environment', provider=InstanceProviderConfig{type='aws'} ...

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.bootstrap.cluster.AddServices: Creating and configuring services [HDFS, YARN, ZOOKEEPER, HIVE, HUE, OOZIE]

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.launchpad.pipeline.AbstractJob: Creating cluster services

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.launchpad.pipeline.AbstractJob: Assigning roles to instances

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.bootstrap.cluster.AddServices: Creating 24 roles for service CD-HDFS-yAisuTBh

[2016-04-20 09:06:05] ERROR [pipeline-thread-51] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed

com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RolesResource.createRoles",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"}

at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:319) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:98) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]

at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]

at sun.reflect.GeneratedMethodAccessor133.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]

at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$fe51a31f.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]

at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]

at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_71]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]

at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]

Caused by: com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=RolesResource.createRoles

at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:97) ~[launchpad-cloudera-manager-api-ext-2.0.0.jar!/:2.0.0]

at com.sun.proxy.$Proxy201.createRoles(Unknown Source) ~[na:na]

at com.cloudera.launchpad.bootstrap.cluster.AddServices.manuallyAssignRoles(AddServices.java:402) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:285) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

... 33 common frames omitted

[2016-04-20 09:06:05] ERROR [pipeline-thread-51] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error

com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RolesResource.createRoles",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"}

at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:319) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:98) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]

at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]

at sun.reflect.GeneratedMethodAccessor133.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]

at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]

at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$fe51a31f.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]

at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]

at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_71]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]

at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]

Caused by: com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=RolesResource.createRoles

at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:97) ~[launchpad-cloudera-manager-api-ext-2.0.0.jar!/:2.0.0]

at com.sun.proxy.$Proxy201.createRoles(Unknown Source) ~[na:na]

at com.cloudera.launchpad.bootstrap.cluster.AddServices.manuallyAssignRoles(AddServices.java:402) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:285) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]

... 33 common frames omitted

[2016-04-20 09:06:05] ERROR [pipeline-thread-51] - c.c.l.p.DatabasePipelineRunner: Pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1' failed

at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$fe51a31f

at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerCluster:7

 

[2016-04-20 09:06:05] INFO  [pipeline-thread-51] - c.c.l.p.s.PipelineRepositoryService: Pipeline '9f8c84b8-8f97-48b6-9def-9cd786eefae1': RUNNING -> ERROR

[2016-04-20 09:06:06] INFO  [pipeline-thread-51] - c.c.l.d.ClusterRepositoryService: Cluster 'Client1_DEV_CDH_Cluster': BOOTSTRAPPING -> BOOTSTRAP_FAILED

 

 

Thanks,

 

Damion.

4,347 Views
0 Kudos
1 ACCEPTED SOLUTION

avatar
author-rank ozzielabrat
Rising Star

Created ‎04-21-2016 04:26 PM

Hi,

 

My issue was caused by me incorrectly having multiple of the same "role" on the same instance.

 

ie:  in the configuration file "roles" section I had 2 x NAMENODE's when I only wanted a single NN.

 

I removed the 2nd "NAMENODE" entry and added additional "master_x" groups with specific services/roles and I am now able to create a non-HA CDH 5.7 cluster !

 

Thanks to those who made a comment and assisted me, much appreciated.

 

 

masters_1 {
count: 1

instance: ${instances.d22x} {
tags {
group: MWHAD_Group_1
name: Client1_POC_MWHAD_Instance
}
}

roles {
HDFS: [NAMENODE, NAMENODE, DATANODE, BALANCER, HTTPFS, GATEWAY]
YARN: [NODEMANAGER, GATEWAY]
ZOOKEEPER: [SERVER]
OOZIE: [OOZIE_SERVER]
HUE: [HUE_SERVER]
HIVE: [HIVESERVER2, HIVEMETASTORE, WEBHCAT, GATEWAY]
}
}

 

 

Cheers,

 

Damion.

View solution in original post

4,279 Views
0 Kudos
4 REPLIES 4

avatar
author-rank gvetticaden
Guru

Created ‎04-20-2016 04:49 AM

Need cloudera experts to help answer this.

I too ran into the exact same issue. 

Details are teh following:

  1. Running Cloudera Director 2.0
  2. Trying to Deploy CDH 5 cluster
  3. Cloud Provider: AWS
  4. I followed all the pre-requistes as here: http://www.cloudera.com/documentation/director/latest/topics/director_get_started_aws.html including seting up the security groups to allow inbound and outboudn traffic and also using the right instance types reocmmended by the documetnation.

When I provision a cluster, Cloudera Manager gets installed successfully but hte cluster has boostrap error and fails on task 1763 of 1772.

 

Here is the full exception strack trace:

 

[2016-04-20 02:17:06] ERROR [pipeline-thread-45] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RolesResource.createRoles",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"}
        at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:319) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
        at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:98) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
        at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
        at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
        at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.0.jar!/:3.1.0]
        at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
        at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_77]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
        at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$94ffdb21.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
        at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
        at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
        at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
        at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
        at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_77]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
Caused by: com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=RolesResource.createRoles
        at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:97) ~[launchpad-cloudera-manager-api-ext-2.0.0.jar!/:2.0.0]
        at com.sun.proxy.$Proxy205.createRoles(Unknown Source) ~[na:na]
        at com.cloudera.launchpad.bootstrap.cluster.AddServices.manuallyAssignRoles(AddServices.java:402) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
        at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:285) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
        ... 33 common frames omitted
[2016-04-20 02:17:06] ERROR [pipeline-thread-45] - c.c.l.p.DatabasePipelineRunner: Pipeline '89241380-0405-4a70-8466-abd689af2cde' failed
        at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$94ffdb21
        at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerCluster:7

 

4,330 Views
0 Kudos

avatar
R.Peter author-rank
Cloudera Employee

Created ‎04-21-2016 07:42 AM

Hi gvetticaden,

 

I have seen the same issue when I tried to deploy 2 NameNodes with 2 master role. If you would like to create a HA, please check this doc: http://www.cloudera.com/documentation/director/latest/topics/director_create_ha_clusters.html

 

4,294 Views
0 Kudos

avatar
author-rank ozzielabrat
Rising Star

Created ‎04-21-2016 04:26 PM

Hi,

 

My issue was caused by me incorrectly having multiple of the same "role" on the same instance.

 

ie:  in the configuration file "roles" section I had 2 x NAMENODE's when I only wanted a single NN.

 

I removed the 2nd "NAMENODE" entry and added additional "master_x" groups with specific services/roles and I am now able to create a non-HA CDH 5.7 cluster !

 

Thanks to those who made a comment and assisted me, much appreciated.

 

 

masters_1 {
count: 1

instance: ${instances.d22x} {
tags {
group: MWHAD_Group_1
name: Client1_POC_MWHAD_Instance
}
}

roles {
HDFS: [NAMENODE, NAMENODE, DATANODE, BALANCER, HTTPFS, GATEWAY]
YARN: [NODEMANAGER, GATEWAY]
ZOOKEEPER: [SERVER]
OOZIE: [OOZIE_SERVER]
HUE: [HUE_SERVER]
HIVE: [HIVESERVER2, HIVEMETASTORE, WEBHCAT, GATEWAY]
}
}

 

 

Cheers,

 

Damion.

4,280 Views
0 Kudos

avatar
R.Peter author-rank
Cloudera Employee

Created ‎04-22-2016 07:55 AM

Hello,

 

Great, good to read it!

 

Cheers,

Peter

4,264 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
View More Announcements