Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot activate spark 2.3 on CDH 6.0.1

Cannot activate spark 2.3 on CDH 6.0.1

New Contributor

Hello,

 

Having problem activating spark 2.3 on CDH 6.0.1

 

Some information:

 

python version =2.7.5

scala version = 2.11.8

java version = 1.8.0_131

 

>:[~]: ls -ltr /opt/cloudera/csd/
total 36
-rw-r--r-- 1 cloudera-scm cloudera-scm 15913 2018-06-01 STREAMSETS-3.3.0.jar
-rw-r--r-- 1 cloudera-scm cloudera-scm 19037 10-05 09:45 SPARK2_ON_YARN-2.3.0.cloudera4.jar

 

I restarted cm-server and after then restarted cm-service (via web UI).

 

parcels:

SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179
Distributed

when I press "Activate" I got the following error "CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179." and below you can see part of error log.

 

grep for spark to make sure that do not have any issue with csd file

 

 

>:[/var/log/cloudera-scm-server]: grep -i spark cloudera-scm-server.log                                                                                                                    
2019-02-26 11:23:30,103 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK2_ON_YARN-2.3.0.cloudera4]
2019-02-26 11:23:30,106 INFO main:com.cloudera.parcel.components.ParcelManagerImpl: Adding remote parcel repository : http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera4/
2019-02-26 11:23:32,065 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK-6.0.1]
2019-02-26 11:23:32,076 INFO main:com.cloudera.parcel.components.ParcelManagerImpl: Adding remote parcel repository : https://archive.cloudera.com/spark/parcels/latest/
2019-02-26 11:23:32,087 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN53-6.0.1]
2019-02-26 11:23:32,095 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN54-6.0.1]
2019-02-26 11:23:32,102 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_10-6.0.1]
2019-02-26 11:23:32,163 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_11-6.0.1]
2019-02-26 11:23:32,221 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN6_0-6.0.1]
2019-02-26 11:23:51,549 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el7.parcel,CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel,CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel,CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel,KUDU-1.4.0-1.cdh5.12.0.p0.25-el7.parcel,STREAMSETS_DATACOLLECTOR-3.3.0-el7.parcel,SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel,CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel under /opt/cloudera/parcel-repo
2019-02-26 11:25:43,146 INFO scm-web-105:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases (  ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.3.p0.611179} is activated.
2019-02-26 11:25:44,508 INFO scm-web-112:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 11:25:44,582 WARN scm-web-112:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.
2019-02-26 11:35:09,949 INFO scm-web-108:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases (  ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.3.p0.611179} is activated.
2019-02-26 11:35:11,468 INFO scm-web-108:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 11:35:11,509 WARN scm-web-108:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.
2019-02-26 11:54:51,411 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: (32 skipped) Found files SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el7.parcel,CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel,CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel,CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel,KUDU-1.4.0-1.cdh5.12.0.p0.25-el7.parcel,STREAMSETS_DATACOLLECTOR-3.3.0-el7.parcel,SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel,CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel under /opt/cloudera/parcel-repo
2019-02-26 12:00:57,718 INFO scm-web-103:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases (  ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.3.p0.611179} is activated.
2019-02-26 12:00:59,366 INFO scm-web-108:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 12:00:59,404 WARN scm-web-108:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.
2019-02-26 12:05:52,387 INFO scm-web-114:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases (  ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.3.p0.611179} is activated.
2019-02-26 12:09:31,396 INFO scm-web-101:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 12:09:31,419 WARN scm-web-101:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.

 

 

Can anyone help? 

 

Part of cloudera-scm-server.log

 

 

2019-02-26 12:00:57,713 INFO scm-web-103:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/1/computeParcelRestart
2019-02-26 12:00:57,718 INFO scm-web-103:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases ( ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.
3.p0.611179} is activated.
2019-02-26 12:00:57,724 INFO scm-web-103:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/1/computeParcelRestart, Status:200
2019-02-26 12:00:57,793 INFO scm-web-115:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/parcel/details
2019-02-26 12:00:57,811 INFO scm-web-115:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/parcel/details, Status:200
2019-02-26 12:00:57,834 INFO scm-web-135:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/parcel/details
2019-02-26 12:00:57,853 INFO scm-web-135:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/parcel/details, Status:200
2019-02-26 12:00:59,361 INFO scm-web-108:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/1/activateParcel
2019-02-26 12:00:59,366 INFO scm-web-108:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 12:00:59,404 WARN scm-web-108:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
2019-02-26 12:00:59,405 ERROR scm-web-108:com.cloudera.server.web.common.JsonResponse: JsonResponse created with throwable: 
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.
at com.cloudera.parcel.components.ParcelDependencyManagerImpl.validateDependencies(ParcelDependencyManagerImpl.java:418)
at com.cloudera.parcel.components.ParcelDependencyManagerImpl.validateDependenciesForActivation(ParcelDependencyManagerImpl.java:375)
at com.cloudera.parcel.components.ParcelManagerImpl.activateParcel(ParcelManagerImpl.java:295)
at com.cloudera.parcel.components.ParcelManagerImpl.activateParcel(ParcelManagerImpl.java:246)
at com.cloudera.server.web.cmf.parcel.ParcelActivationController.activateParcel(ParcelActivationController.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:200)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:158)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:134)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
at java.lang.Thread.run(Thread.java:748)
2019-02-26 12:00:59,407 INFO scm-web-108:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/1/activateParcel, Status:200

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here