CDH 6.3 parcels fail to activate during Altus 6.3 cluster bootstrap on AWS

[#1]

Altus Director 6.3

Bootstrap of CDH6.3 clusters are failing due to failed activation of parcels. We've successfully deployed earlier versions of CDH 6.x.  On all nodes the agent logs show the following error for the same parcel:

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'spark', '-K', 'UMASK=022', '--home', '/var/lib/spark', '--comment', 'Spark', '--shell', '/sbin/nologin', 'spark']

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     Ensuring correct file permissions for new parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813.

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     chown: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db hue hue

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     Executing command ['chown', 'hue:hue', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db']

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     chmod: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db 0660

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     Executing command ['chmod', '0660', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop/desktop.db']

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     chown: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop hue hue

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     Executing command ['chown', 'hue:hue', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop']

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     chmod: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop 0755

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     Executing command ['chmod', '0755', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hue/desktop']

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     chown: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-yarn/bin/container-executor root yarn

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     Executing command ['chown', 'root:yarn', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-yarn/bin/container-executor']

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     chmod: /opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-yarn/bin/container-executor 6050

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       INFO     Executing command ['chmod', '6050', u'/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-yarn/bin/container-executor']

[14/Aug/2019 22:45:18 +0000] 31970 MainThread parcel       ERROR    Error while attempting to modify permissions of file '/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'.

Traceback (most recent call last):

  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel.py", line 520, in ensure_permissions

    file = cmf.util.validate_and_open_fd(path, self.get_parcel_home(parcel))

  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/util/__init__.py", line 358, in validate_and_open_fd

    fd = os.open(path, flags)

OSError: [Errno 2] No such file or directory: '/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'

Altus director server log shows:

2019-08-14 22:45:35.535 +0000] ERROR [p-b96f4ed12bad-DefaultBootstrapClusterJob] 3812e6e2-ba96-42c5-b53e-4cd2e99a54de POST /api/d6.2/environments/ez2/deployments/ez2/clusters com.cloudera.launchpad.bootstrap.cluster.ActivateParcel - com.cloudera.launchpad.pipeline.util.PipelineRunner: Attempt to execute job failed
com.cloudera.api.ext.ClouderaManagerException: API call to Cloudera Manager failed.

 Method=ParcelResource.activateCommand. Response Status Code: 400. Message: {
"message" : "The version change by activating the parcel would cause an error"

[#2]

Could you please check for errors on the Cloudera Manager server log?

[#3]

The log entry from Altus Director I provided mirrors the error I saw on the
manager node while tailing the logs there. Parcel distribution /
activation seemed to be going fine until it hit the error I posted.
Unfortunately I've terminated the deployment as this was our 4th attempt.
I was thinking of pinning CDH version in the cluster bootstrap to 6.2
instead of 6 so that it doesn't grab the latest (6.3) parcels. If I hit
the same issue, I will rerun my original config and provide more logs from
the manager node.

[#4]

I tried another deployment.  Here are some entires that standout in the manager server log - at least to me:

2019-08-15 00:09:29,753 WARN scm-web-93:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,754 INFO scm-web-93:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.158 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:29,755 WARN scm-web-177:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,757 INFO scm-web-177:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.246 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:29,771 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,774 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.244 over PT0.003S. Rate: 25.352 MB/s
2019-08-15 00:09:29,785 WARN scm-web-161:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,785 INFO scm-web-161:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.191 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:29,819 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,819 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.215 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:29,827 WARN scm-web-93:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,827 INFO scm-web-93:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.226 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:29,868 WARN scm-web-91:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,869 INFO scm-web-91:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.208 over PT0.001S. Rate: 76.056 MB/s
2019-08-15 00:09:29,874 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,874 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.250 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:29,874 WARN scm-web-290:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,875 INFO scm-web-290:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.241 over PT0.001S. Rate: 76.056 MB/s
2019-08-15 00:09:29,905 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,905 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.243 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:29,914 WARN scm-web-170:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,915 INFO scm-web-170:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.137 over PT0.001S. Rate: 76.056 MB/s
2019-08-15 00:09:29,920 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,925 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.185 over PT0.001S. Rate: 76.056 MB/s
2019-08-15 00:09:29,928 WARN scm-web-177:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,929 INFO scm-web-177:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.138 over PT0.001S. Rate: 76.056 MB/s
2019-08-15 00:09:29,993 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,993 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.165 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:29,993 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:29,994 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.149 over PT0.001S. Rate: 76.056 MB/s
2019-08-15 00:09:30,018 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:30,018 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.150 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:30,030 WARN scm-web-94:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:30,030 INFO scm-web-94:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.188 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:30,034 INFO scm-web-89:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.24.1.165
2019-08-15 00:09:30,048 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent
2019-08-15 00:09:30,048 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.247 over PT0S. Rate: Infinity MB/s
2019-08-15 00:09:30,065 WARN scm-web-94:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent

Then there are several INFO entries which seem to indicate parcels are being distributed / activated successfully until we hit the following error, after which Altus Director fails the deployment:

/clusters/ez2/parcels/products/CDH/versions/6.3.0-1.cdh6.3.0.p0.1279813/commands/activate
2019-08-15 00:12:19,738 INFO scm-web-290:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel CDH:6.3.0-1.cdh6.3.0.p0.1279813 on cluster ez2
2019-08-15 00:12:19,756 INFO scm-web-290:com.cloudera.enterprise.AbstractWrappedEntityManager: Rolling back transaction that wasn't marked for rollback-only.
java.lang.Exception: Non-thrown exception for stack trace.
at com.cloudera.enterprise.AbstractWrappedEntityManager.close(AbstractWrappedEntityManager.java:161)
at com.cloudera.cmf.persist.CmfEntityManager.close(CmfEntityManager.java:367)
at com.cloudera.server.web.cmf.AuthFilterEntityManager.close(AuthFilterEntityManager.java:191)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:209)
at com.cloudera.api.dao.impl.ManagerDaoBase.access$100(ManagerDaoBase.java:82)
at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:239)
at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210)
at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:131)
at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:169)
at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:272)
at com.sun.proxy.$Proxy205.activate(Unknown Source)
at com.cloudera.api.v3.impl.ParcelResourceImpl.activateCommand(ParcelResourceImpl.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261)
at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:263)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:189)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:218)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:200)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.cloudera.api.ApiBasicAuthFilter.doFilter(ApiBasicAuthFilter.java:86)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at com.cloudera.server.common.BoundedQueuedThreadPool$2.run(BoundedQueuedThreadPool.java:94)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
2019-08-15 00:12:19,759 INFO scm-web-290:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation. Msg:The version change by activating the parcel would cause an error
java.lang.IllegalArgumentException: The version change by activating the parcel would cause an error

[#5]

Are you using a Director template to create the deployment or is this being launched from the UI?

If you are using a template, would it be possible to share it?

[#6]

We use templates to deploy our clusters.  I've discovered why the parcel activation is failing.  Version 6.2 agents are being installed while version 6.3 parcels are being distributed / activated.  This same template worked fine when 6 was originally released.  In the template we have:

cluster {
    products {
      CDH: 6

We are not defining a parcel URL, which I assume means we are using the default based on the version of director, which is 6.3.0.  I've tried defining more specific version of CDH above (6.2, 6.3) but director continues to deploy 6.2 agents and 6.3 parcels.

[#7]

Any suggestions for why director is deploying conflicting versions of the agent and parcels?  Seems like a fundamental issue. 

[#8 · Accepted solution]

OK - so it seems that the yum repo for cloudera manager was defined / pinned to 6.2 in the cluster definition - so of course Director did what it was being asked to do.   Updated to point to 6.3.

repository: "https://archive.cloudera.com/cm6/6.3/redhat7/yum/"
repositoryKeyUrl: "https://archive.cloudera.com/cm6/6.3/redhat7/yum/RPM-GPG-KEY-cloudera"