Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark role is creating issue

Highlighted

Spark role is creating issue

Explorer

First I was getting  below error

Error 1:

Command failed to run because service HIVE-1 has invalid configuration. Review and correct its configuration.
First error: All HiveServer2 roles must have a Spark role (such as a Gateway) on the same host to pick up the Spark configuration.
You must deploy Spark client configuration before this will work correctly. The following hosts have a HiveServer2 but no Spark role:
ip-10-0-1-88.us-west-2.compute.internal, ip-10-0-1-89.us-west-2.compute.internal., serviceRef=null, roleRef=null, hostRef=null, parent=null}

 

I added below Role to masters-1 :

SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]

 

Now i am getting Below errors :

[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> CreateList/1 [[]]
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: << DatabaseValue{delegate=PersistentValueEntity{id=4225, pipeline=3ea8a5cc-1208-4984-beca-fe1b8657f324, ...
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: >> AddServices/5 [Environment{name='C5-HAPerfEnvUL-AWS Environment', provider=InstanceProviderConfig{type='aws'}, cre ...
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating and configuring services [HDFS, YARN, ZOOKEEPER, HBASE, HIVE, HUE, OOZIE, SENTRY, SPARK_ON_YARN, SOLR, KS_INDEXER, HBASERESTSERVER, HBASETHRIFTSERVER]
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Invalid service type(s) specified. Ignored during service and role creation: HBASERESTSERVER,HBASETHRIFTSERVER
[2016-06-11 03:54:51] INFO [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Creating cluster services
[2016-06-11 03:54:53] INFO [pipeline-thread-1] - c.c.launchpad.pipeline.AbstractJob: Assigning roles to instances
[2016-06-11 03:54:53] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 10 roles for service CD-HDFS-HClQrKLR
[2016-06-11 03:54:53] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 5 roles for service CD-YARN-mNbdfxMn
[2016-06-11 03:54:53] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 3 roles for service CD-ZOOKEEPER-HXoaUHKZ
[2016-06-11 03:54:54] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 9 roles for service CD-HBASE-tYMchGdW
[2016-06-11 03:54:54] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 5 roles for service CD-HIVE-ArVFfpAx
[2016-06-11 03:54:54] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 2 roles for service CD-HUE-pXFOLFAH
[2016-06-11 03:54:55] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 2 roles for service CD-OOZIE-uVJoqAJQ
[2016-06-11 03:54:55] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 1 roles for service CD-SENTRY-PAuEysaU
[2016-06-11 03:54:55] INFO [pipeline-thread-1] - c.c.l.bootstrap.cluster.AddServices: Creating 3 roles for service CD-SPARK_ON_YARN-xPhUGdAJ
[2016-06-11 03:54:55] ERROR [pipeline-thread-1] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RolesResource.createRoles",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"}
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:319) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:98) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.2.jar!/:3.1.2]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$3494b11c.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) [launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) [launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=RolesResource.createRoles
at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:97) ~[launchpad-cloudera-manager-api-ext-2.0.0.jar!/:2.0.0]
at com.sun.proxy.$Proxy175.createRoles(Unknown Source) ~[na:na]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.manuallyAssignRoles(AddServices.java:402) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:285) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
... 33 common frames omitted
[2016-06-11 03:54:55] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Encountered an unrecoverable error
com.cloudera.launchpad.pipeline.UnrecoverablePipelineError: ClouderaManagerException{message="API call to Cloudera Manager failed. Method=RolesResource.createRoles",causeClass=class javax.ws.rs.BadRequestException, causeMessage="null"}
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:319) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:98) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job5.runUnchecked(Job5.java:34) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.job.Job5$$FastClassBySpringCGLIB$$54178505.invoke(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.codahale.metrics.Timer.time(Timer.java:101) ~[metrics-core-3.1.2.jar!/:3.1.2]
at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) ~[spring-aop-4.1.6.RELEASE.jar!/:4.1.6.RELEASE]
at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$3494b11c.runUnchecked(<generated>) ~[spring-core-4.1.6.RELEASE.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:159) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:130) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:99) ~[launchpad-pipeline-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:125) ~[launchpad-pipeline-database-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) [launchpad-common-2.0.0.jar!/:2.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed. Method=RolesResource.createRoles
at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:97) ~[launchpad-cloudera-manager-api-ext-2.0.0.jar!/:2.0.0]
at com.sun.proxy.$Proxy175.createRoles(Unknown Source) ~[na:na]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.manuallyAssignRoles(AddServices.java:402) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
at com.cloudera.launchpad.bootstrap.cluster.AddServices.run(AddServices.java:285) ~[launchpad-bootstrap-2.0.0.jar!/:2.0.0]
... 33 common frames omitted
[2016-06-11 03:54:55] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Pipeline '3ea8a5cc-1208-4984-beca-fe1b8657f324' failed
at com.cloudera.launchpad.bootstrap.cluster.AddServices$$EnhancerBySpringCGLIB$$3494b11c
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerCluster:7

[2016-06-11 03:54:55] INFO [pipeline-thread-1] - c.c.l.p.s.PipelineRepositoryService: Pipeline '3ea8a5cc-1208-4984-beca-fe1b8657f324': RUNNING -> ERROR
[2016-06-11 03:55:01] INFO [pipeline-thread-1] - c.c.l.d.ClusterRepositoryService: Cluster 'C5-HAPerfEnvUL-AWS': BOOTSTRAPPING -> BOOTSTRAP_FAILED
[2016-06-11 03:55:01] INFO [Thread-2] - o.s.c.a.AnnotationConfigApplicationContext: Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@6f9591b6: startup date [Sat Jun 11 03:17:48 EDT 2016]; root of context hierarchy
[2016-06-11 03:55:01] INFO [Thread-2] - o.s.o.j.LocalContainerEntityManagerFactoryBean: Closing JPA EntityManagerFactory for persistence unit 'default'
[2016-06-11 03:55:01] WARN [Thread-2] - c.c.l.c.security.CipherSchemeFactory: Allowing cipher scheme to be set again, prior scheme is com.cloudera.launchpad.common.security.TripleDESCipher@380e7d73

 

Below are the conf :

 

services: [HDFS, YARN, ZOOKEEPER, HBASE, HIVE, HUE, OOZIE, SENTRY, SPARK_ON_YARN,SOLR,KS_INDEXER]

############ Masters HA start ###########
hdfsmasters-1 {
count: 2
instance: ${instances.i2x} {
tags {
group: hdfsmasters-1
}
}
roles {
HDFS: [NAMENODE, FAILOVERCONTROLLER, JOURNALNODE]
}

# NameNode nameservice, autofailover, and quorum journal name must be configured for high availability
configs {
HDFS {
NAMENODE {
dfs_federation_namenode_nameservice: hanameservice
autofailover_enabled: true
dfs_namenode_quorum_journal_name: hanameservice
}
}
}
}

hdfsmasters-2 {
count: 1
instance: ${instances.i2x} {
tags {
group: hdfsmasters-2
}
}
roles {
HDFS: [JOURNALNODE, HTTPFS] # HTTPFS role needed for HUE
}
}

masters-1 {
count: 2
instance: ${instances.i2x} {
tags {
group: masters-1
}
}
roles {
ZOOKEEPER: [SERVER]
HIVE: [HIVESERVER2, HIVEMETASTORE]
YARN: [RESOURCEMANAGER,JOBHISTORY]
HBASE: [MASTER,HBASETHRIFTSERVER,HBASERESTSERVER]
HUE: [HUE_SERVER]
OOZIE: [OOZIE_SERVER]
SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]
}
# Oozie plugins must be configured for high availability
configs {
OOZIE {
OOZIE_SERVER {
oozie_plugins_list: "org.apache.oozie.service.ZKLocksService,org.apache.oozie.service.ZKXLogStreamingService,org.apache.oozie.service.ZKJobsConcurrencyService,org.apache.oozie.service.ZKUUIDService"
}
}
}
}

masters-2 {
count: 1
instance: ${instances.i2x} {
tags {
group: master
}
}
roles {
ZOOKEEPER: [SERVER]
YARN: [JOBHISTORY]
HBASE: [HBASETHRIFTSERVER] # HBASETHRIFTSERVER role needed for HUE
SOLR: [SOLR_SERVER]
KS_INDEXER: [HBASE_INDEXER]
SENTRY: [SENTRY_SERVER]

}
}
####################### masters end