Hello,

Having problem activating spark 2.3 on CDH 6.0.1

Some information:

python version =2.7.5

scala version = 2.11.8

java version = 1.8.0_131

>:[~]: ls -ltr /opt/cloudera/csd/
total 36
-rw-r--r-- 1 cloudera-scm cloudera-scm 15913 2018-06-01 STREAMSETS-3.3.0.jar
-rw-r--r-- 1 cloudera-scm cloudera-scm 19037 10-05 09:45 SPARK2_ON_YARN-2.3.0.cloudera4.jar

I restarted cm-server and after then restarted cm-service (via web UI).

parcels:

when I press "Activate" I got the following error "CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179." and below you can see part of error log.

grep for spark to make sure that do not have any issue with csd file

>:[/var/log/cloudera-scm-server]: grep -i spark cloudera-scm-server.log                                                                                                                    
2019-02-26 11:23:30,103 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK2_ON_YARN-2.3.0.cloudera4]
2019-02-26 11:23:30,106 INFO main:com.cloudera.parcel.components.ParcelManagerImpl: Adding remote parcel repository : http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera4/
2019-02-26 11:23:32,065 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK-6.0.1]
2019-02-26 11:23:32,076 INFO main:com.cloudera.parcel.components.ParcelManagerImpl: Adding remote parcel repository : https://archive.cloudera.com/spark/parcels/latest/
2019-02-26 11:23:32,087 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN53-6.0.1]
2019-02-26 11:23:32,095 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN54-6.0.1]
2019-02-26 11:23:32,102 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_10-6.0.1]
2019-02-26 11:23:32,163 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN5_11-6.0.1]
2019-02-26 11:23:32,221 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK_ON_YARN6_0-6.0.1]
2019-02-26 11:23:51,549 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el7.parcel,CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel,CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel,CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel,KUDU-1.4.0-1.cdh5.12.0.p0.25-el7.parcel,STREAMSETS_DATACOLLECTOR-3.3.0-el7.parcel,SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel,CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel under /opt/cloudera/parcel-repo
2019-02-26 11:25:43,146 INFO scm-web-105:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases (  ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.3.p0.611179} is activated.
2019-02-26 11:25:44,508 INFO scm-web-112:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 11:25:44,582 WARN scm-web-112:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.
2019-02-26 11:35:09,949 INFO scm-web-108:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases (  ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.3.p0.611179} is activated.
2019-02-26 11:35:11,468 INFO scm-web-108:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 11:35:11,509 WARN scm-web-108:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.
2019-02-26 11:54:51,411 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: (32 skipped) Found files SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el7.parcel,CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel,CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel,CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel,KUDU-1.4.0-1.cdh5.12.0.p0.25-el7.parcel,STREAMSETS_DATACOLLECTOR-3.3.0-el7.parcel,SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179-el7.parcel,CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel under /opt/cloudera/parcel-repo
2019-02-26 12:00:57,718 INFO scm-web-103:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases (  ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.3.p0.611179} is activated.
2019-02-26 12:00:59,366 INFO scm-web-108:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 12:00:59,404 WARN scm-web-108:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.
2019-02-26 12:05:52,387 INFO scm-web-114:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases (  ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.3.p0.611179} is activated.
2019-02-26 12:09:31,396 INFO scm-web-101:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 12:09:31,419 WARN scm-web-101:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.

Can anyone help? 

Part of cloudera-scm-server.log

2019-02-26 12:00:57,713 INFO scm-web-103:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/1/computeParcelRestart
2019-02-26 12:00:57,718 INFO scm-web-103:com.cloudera.parcel.components.ParcelManagerImpl: Determined obsoleted releases ( ) if release DbRelease{id=54, product=SPARK2, version=2.3.0.cloudera4-1.cdh5.13.
3.p0.611179} is activated.
2019-02-26 12:00:57,724 INFO scm-web-103:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/1/computeParcelRestart, Status:200
2019-02-26 12:00:57,793 INFO scm-web-115:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/parcel/details
2019-02-26 12:00:57,811 INFO scm-web-115:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/parcel/details, Status:200
2019-02-26 12:00:57,834 INFO scm-web-135:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/parcel/details
2019-02-26 12:00:57,853 INFO scm-web-135:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/parcel/details, Status:200
2019-02-26 12:00:59,361 INFO scm-web-108:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/1/activateParcel
2019-02-26 12:00:59,366 INFO scm-web-108:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 on cluster cluster
2019-02-26 12:00:59,404 WARN scm-web-108:com.cloudera.server.web.cmf.WebController: Failed to activate parcel SPARK2:2.3.0.cloudera4-1.cdh5.13.3.p0.611179 for cluster DbCluster{id=1, name=cluster}
2019-02-26 12:00:59,405 ERROR scm-web-108:com.cloudera.server.web.common.JsonResponse: JsonResponse created with throwable: 
com.cloudera.parcel.ParcelRelationsException: CDH 6.0.1-1.cdh6.0.1.p0.590678 replaces SPARK2 2.3.0.cloudera4-1.cdh5.13.3.p0.611179.
at com.cloudera.parcel.components.ParcelDependencyManagerImpl.validateDependencies(ParcelDependencyManagerImpl.java:418)
at com.cloudera.parcel.components.ParcelDependencyManagerImpl.validateDependenciesForActivation(ParcelDependencyManagerImpl.java:375)
at com.cloudera.parcel.components.ParcelManagerImpl.activateParcel(ParcelManagerImpl.java:295)
at com.cloudera.parcel.components.ParcelManagerImpl.activateParcel(ParcelManagerImpl.java:246)
at com.cloudera.server.web.cmf.parcel.ParcelActivationController.activateParcel(ParcelActivationController.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:200)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:158)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:121)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:134)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
at java.lang.Thread.run(Thread.java:748)
2019-02-26 12:00:59,407 INFO scm-web-108:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/1/activateParcel, Status:200