Reply
Highlighted
Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Spark role is creating issue

First I was getting  below error

Error 1:

Command failed to run because service HIVE-1 has invalid configuration. Review and correct its configuration.
First error: All HiveServer2 roles must have a Spark role (such as a Gateway) on the same host to pick up the Spark configuration.
You must deploy Spark client configuration before this will work correctly. The following hosts have a HiveServer2 but no Spark role:
ip-10-0-1-88.us-west-2.compute.internal, ip-10-0-1-89.us-west-2.compute.internal., serviceRef=null, roleRef=null, hostRef=null, parent=null}

 

I added below Role to masters-1 :

SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]

 

Now i am getting Below errors :

[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> CreateList/1 [[]]
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=4225, pipeline=3ea8a5cc-1208-4984-beca-fe1b8657f324, ...
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> AddServices/5 [Environment{name='C5-HAPerfEnvUL-AWS Environment', provider=InstanceProviderConfig{type='aws'}, cre ...
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating and configuring services [HDFS, YARN, ZOOKEEPER, HBASE, HIVE, HUE, OOZIE, SENTRY, SPARK_ON_YARN, SOLR, KS_INDEXER, HBASERESTSERVER, HBASETHRIFTSERVER]
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Invalid service type(s) specified. Ignored during service and role creation: HBASERESTSERVER,HBASETHRIFTSERVER
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Creating cluster services
[2016-06-11 03:54:53] INFO [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Assigning roles to instances
[2016-06-11 03:54:53] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 10 roles for service CD-HDFS-HClQrKLR
[2016-06-11 03:54:53] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 5 roles for service CD-YARN-mNbdfxMn
[2016-06-11 03:54:53] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 3 roles for service CD-ZOOKEEPER-HXoaUHKZ
[2016-06-11 03:54:54] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 9 roles for service CD-HBASE-tYMchGdW
[2016-06-11 03:54:54] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 5 roles for service CD-HIVE-ArVFfpAx
[2016-06-11 03:54:54] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 2 roles for service CD-HUE-pXFOLFAH
[2016-06-11 03:54:55] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 2 roles for service CD-OOZIE-uVJoqAJQ
[2016-06-11 03:54:55] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 1 roles for service CD-SENTRY-PAuEysaU
[2016-06-11 03:54:55] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 3 roles for service CD-SPARK_ON_YARN-xPhUGdAJ
[2016-06-11 03:54:55] ERROR [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RolesResource.createRoles",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"}
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:319) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:98) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.2.jar!/:3.1.2]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$3494b11c.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=RolesResource.createRoles
at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:97) ~[launchpad-cloudera-manager-api-ext-2.0.0.jar!/:2.0.0]
at com.sun.proxy.$Proxy175.createRoles(Unknown Source) ~[na:na]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.manuallyAssignRoles(AddServices.java:402) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:285) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
... 33 common frames omitted
[2016-06-11 03:54:55] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RolesResource.createRoles",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"}
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:319) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:98) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.2.jar!/:3.1.2]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$3494b11c.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=RolesResource.createRoles
at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:97) ~[launchpad-cloudera-manager-api-ext-2.0.0.jar!/:2.0.0]
at com.sun.proxy.$Proxy175.createRoles(Unknown Source) ~[na:na]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.manuallyAssignRoles(AddServices.java:402) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:285) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
... 33 common frames omitted
[2016-06-11 03:54:55] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Pipeline '3ea8a5cc-1208-4984-beca-fe1b8657f324' failed
at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$3494b11c
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerCluster:7

[2016-06-11 03:54:55] INFO [pipeline-thread-1] - c.c.l.p.s.PipelineRepositoryService: Pipeline '3ea8a5cc-1208-4984-beca-fe1b8657f324': RUNNING -> ERROR
[2016-06-11 03:55:01] INFO [pipeline-thread-1] - c.c.l.d.ClusterRepositoryService: Cluster 'C5-HAPerfEnvUL-AWS': BOOTSTRAPPING -> BOOTSTRAP_FAILED
[2016-06-11 03:55:01] INFO [Thread-2] - o.s.c.a.AnnotationConfigApplicationContext: Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@6f9591b6: startup date [Sat Jun 11 03:17:48 EDT 2016]; root of context hierarchy
[2016-06-11 03:55:01] INFO [Thread-2] - o.s.o.j.LocalContainerEntityManagerFactoryBean: Closing JPA EntityManagerFactory for persistence unit 'default'
[2016-06-11 03:55:01] WARN [Thread-2] - c.c.l.c.security.CipherSchemeFactory: Allowing cipher scheme to be set again, prior scheme is com.cloudera.launchpad.common.security.TripleDESCipher@380e7d73

 

Below are the conf :

 

services: [HDFS, YARN, ZOOKEEPER, HBASE, HIVE, HUE, OOZIE, SENTRY, SPARK_ON_YARN,SOLR,KS_INDEXER]

############ Masters HA start ###########
hdfsmasters-1 {
count: 2
instance: ${instances.i2x} {
tags {
group: hdfsmasters-1
}
}
roles {
HDFS: [NAMENODE, FAILOVERCONTROLLER, JOURNALNODE]
}

# NameNode nameservice, autofailover, and quorum journal name must be configured for high availability
configs {
HDFS {
NAMENODE {
dfs_federation_namenode_nameservice: hanameservice
autofailover_enabled: true
dfs_namenode_quorum_journal_name: hanameservice
}
}
}
}

hdfsmasters-2 {
count: 1
instance: ${instances.i2x} {
tags {
group: hdfsmasters-2
}
}
roles {
HDFS: [JOURNALNODE, HTTPFS] # HTTPFS role needed for HUE
}
}

masters-1 {
count: 2
instance: ${instances.i2x} {
tags {
group: masters-1
}
}
roles {
ZOOKEEPER: [SERVER]
HIVE: [HIVESERVER2, HIVEMETASTORE]
YARN: [RESOURCEMANAGER,JOBHISTORY]
HBASE: [MASTER,HBASETHRIFTSERVER,HBASERESTSERVER]
HUE: [HUE_SERVER]
OOZIE: [OOZIE_SERVER]
SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]
}
# Oozie plugins must be configured for high availability
configs {
OOZIE {
OOZIE_SERVER {
oozie_plugins_list: "org.apache.oozie.service.ZKLocksService,org.apache.oozie.service.ZKXLogStreamingService,org.apache.oozie.service.ZKJobsConcurrencyService,org.apache.oozie.service.ZKUUIDService"
}
}
}
}

masters-2 {
count: 1
instance: ${instances.i2x} {
tags {
group: master
}
}
roles {
ZOOKEEPER: [SERVER]
YARN: [JOBHISTORY]
HBASE: [HBASETHRIFTSERVER] # HBASETHRIFTSERVER role needed for HUE
SOLR: [SOLR_SERVER]
KS_INDEXER: [HBASE_INDEXER]
SENTRY: [SENTRY_SERVER]

}
}
####################### masters end

 

Announcements