Apache recently posted an update to the existing log4j2 vulnerabilities already discussed here.    From the Apache page:   " Apache Log4j2 versions 2.0-alpha1 through 2.16.0, excluding 2.12.3, did not protect from uncontrolled recursion from self-referential lookups. When the logging configuration uses a non-default Pattern Layout with a Context Lookup (for example, $${ctx:loginId}), attackers with control over Thread Context Map (MDC) input data can craft malicious input data that contains a recursive lookup, resulting in a StackOverflowError that will terminate the process. This is also known as a DOS (Denial of Service) attack."   My company runs several private CDH6.3 clusters and we've already applied the original log4j2 patch from https://github.com/cloudera/cloudera-scripts-for-log4j.   Can anyone confirm if CDH products are susceptible to CVE-2021-45105?  Seems this particular vulnerability is enabled only if logging configuration uses a non-default pattern layout with a context lookup.  The mitigation strategy differs from that taken by the existing log4j2 patch from Cloudera.     Apache log4j security vulnerabilities: https://logging.apache.org/log4j/2.x/security.html#CVE-2021-45105   ... View more

There was a recent update from Apache; https://logging.apache.org/log4j/2.x/security.html#CVE-2021-45105    "Apache Log4j2 does not always protect from infinite recursion in lookup evaluation", which seems to imply the mitigation strategy of removing references to jndilookup class will not address this.   Is this a concern for CDH6.3?  If so, will the patch script be updated to address? ... View more

Answered my own question.  I mistakingly thought distcp2 supported the -direct copy option but it does not.   ... View more

I'm running CDH 6.3 ( Hadoop 3.0.0 and MR2 ).   If what you are saying is true, then shouldn't I be able to pass the "-direct" option to my hadoop distcp commands?   Currently it fails to recognize this command line option, which tells me distcp2 isn't being used. ... View more

Regarding the updated paywall policy - specifically the following comment:   " Furthermore, when a license expires, users will no longer be able to access the Cloudera Manager Admin console to manage clusters until a valid license is uploaded. "   We have several unlicensed clusters running.  Do you interpret unlicensed clusters as clusters with expired licenses, in which case we stand to lose access to our Cloudera Manager consoles at any time now? ... View more

Unbelievable. ... View more

Some additional information... We use Packer to build our images, and apply RedHat's CIS security policy for compliance reasons which sets a more restrictive umask in /etc/bashrc if [ $UID -gt 199 ] && [ “/usr/bin/id -gn” = “/usr/bin/id -un” ]; then umask 027 else umask 022 fi I'm thinking this doesn't effect our CDH5 deployments because those images already have the correct package version installed, however during CDH6 bootstrap, updated version are required and are installed using 027 umask resulting in no permissions for non-root user.  Does the cluster bootstrap process assume umask 022 for non-root users?    ... View more

Any suggestions for why director is deploying conflicting versions of the agent and parcels?  Seems like a fundamental issue.  ... View more

We use templates to deploy our clusters.  I've discovered why the parcel activation is failing.  Version 6.2 agents are being installed while version 6.3 parcels are being distributed / activated.  This same template worked fine when 6 was originally released.  In the template we have: cluster { products { CDH: 6 We are not defining a parcel URL, which I assume means we are using the default based on the version of director, which is 6.3.0.  I've tried defining more specific version of CDH above (6.2, 6.3) but director continues to deploy 6.2 agents and 6.3 parcels. ... View more

I tried another deployment.  Here are some entires that standout in the manager server log - at least to me:   2019-08-15 00:09:29,753 WARN scm-web-93:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,754 INFO scm-web-93:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.158 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,755 WARN scm-web-177:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,757 INFO scm-web-177:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.246 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,771 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,774 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.244 over PT0.003S. Rate: 25.352 MB/s 2019-08-15 00:09:29,785 WARN scm-web-161:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,785 INFO scm-web-161:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.191 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,819 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,819 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.215 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,827 WARN scm-web-93:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,827 INFO scm-web-93:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.226 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,868 WARN scm-web-91:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,869 INFO scm-web-91:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.208 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,874 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,874 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.250 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,874 WARN scm-web-290:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,875 INFO scm-web-290:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.241 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,905 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,905 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.243 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,914 WARN scm-web-170:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,915 INFO scm-web-170:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.137 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,920 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,925 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.185 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,928 WARN scm-web-177:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,929 INFO scm-web-177:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.138 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,993 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,993 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.165 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,993 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,994 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.149 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:30,018 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:30,018 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.150 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:30,030 WARN scm-web-94:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:30,030 INFO scm-web-94:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.188 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:30,034 INFO scm-web-89:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.24.1.165 2019-08-15 00:09:30,048 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:30,048 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.247 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:30,065 WARN scm-web-94:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent   Then there are several INFO entries which seem to indicate parcels are being distributed / activated successfully until we hit the following error, after which Altus Director fails the deployment:     /clusters/ez2/parcels/products/CDH/versions/6.3.0-1.cdh6.3.0.p0.1279813/commands/activate 2019-08-15 00:12:19,738 INFO scm-web-290:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel CDH:6.3.0-1.cdh6.3.0.p0.1279813 on cluster ez2 2019-08-15 00:12:19,756 INFO scm-web-290:com.cloudera.enterprise.AbstractWrappedEntityManager: Rolling back transaction that wasn't marked for rollback-only. java.lang.Exception: Non-thrown exception for stack trace. at com.cloudera.enterprise.AbstractWrappedEntityManager.close(AbstractWrappedEntityManager.java:161) at com.cloudera.cmf.persist.CmfEntityManager.close(CmfEntityManager.java:367) at com.cloudera.server.web.cmf.AuthFilterEntityManager.close(AuthFilterEntityManager.java:191) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:209) at com.cloudera.api.dao.impl.ManagerDaoBase.access$100(ManagerDaoBase.java:82) at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:239) at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32) at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210) at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:131) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:169) at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:272) at com.sun.proxy.$Proxy205.activate(Unknown Source) at com.cloudera.api.v3.impl.ParcelResourceImpl.activateCommand(ParcelResourceImpl.java:70) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99) at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:263) at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:189) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:218) at javax.servlet.http.HttpServlet.service(HttpServlet.java:665) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:200) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at com.cloudera.api.ApiBasicAuthFilter.doFilter(ApiBasicAuthFilter.java:86) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at com.cloudera.server.common.BoundedQueuedThreadPool$2.run(BoundedQueuedThreadPool.java:94) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) 2019-08-15 00:12:19,759 INFO scm-web-290:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation. Msg:The version change by activating the parcel would cause an error java.lang.IllegalArgumentException: The version change by activating the parcel would cause an error   ... View more

The log entry from Altus Director I provided mirrors the error I saw on the manager node while tailing the logs there. Parcel distribution / activation seemed to be going fine until it hit the error I posted. Unfortunately I've terminated the deployment as this was our 4th attempt. I was thinking of pinning CDH version in the cluster bootstrap to 6.2 instead of 6 so that it doesn't grab the latest (6.3) parcels. If I hit the same issue, I will rerun my original config and provide more logs from the manager node. ... View more

Thank you. Exactly what I was looking for. ... View more

How do I navigate to all my posts in the new layout? ... View more

I'm in the middle of another deployment where I see random failures reported by director when attempting to connect to port 22 of cluster instances during bootstrap.  I'm seeing this for deployments where I haven't enabled publicip - so it's no just for publicip enabled deployments.  Each time, it's only 1 or 2 of the instances and each time I can connect to the instance from the director server just fine, even during the retry window when director still claims it cannot connect.  Not sure why director thinks it can't connect when I clearly can from the same server.  I've confirmed the forward/reverse DNS resolution is working.  I basically have to deploy several times before I hit the sweet spot and the deployment completes.  This type of inconsistent behavior has been systemic with Azure deployments.  I've experienced none of this with AWS deployments.  I'll attempt to collect more logs. ... View more

Thanks for your reply.   Bootstrap did fail in this case, which surprised me given the fact that during the retry window I could ssh to the instance's private IP from Director.  Also, Director had no problem connecting to the other 2 master nodes that I enabled public IP on.  The log entries I provided are repeated over and over until the retries are exhausted.   ... View more

Just checking in to see if there are any updates.   As a workaround we are using Azure MySQL backend, but we would prefer to use Azure Postgresql.  Let me know if I can provide any additional info.  Unfortunately it's a very repeatable issue.  🙂 ... View more

Thank you for the clarification.      D. ... View more

Thank you for the response.   To your point about bootstrapping/terminating - I have noticed during testing when I'm cycling through many bootstrapping / terminations that Director can get into a state where it seems to no longer respond to bootstrapping requests (from command line).    I'll typically let Director "cool down" for a few minutes and retry.   ... View more

Cleaned up the redundant  common-instanceTemplate section per the latest spec.  Looks like Director is now accepting my custom image - however now I'm running into the other limitation that's been a thorn in my side, which is the fact that the custom image can't have data disks defined like they can in AWS.   I believe the workaround for that is to define disk count as 0, but I'll need to rebuild my worker packer image to include the additional data disks for hdfs.  Anyway, different subject.   Thank you Bill for the guidance.  I'll mark this issue as resolved.  ... View more

Bill, I belived I misinterpreted your initial suggestion.  I was focused on the common-instanceTemplate section entirely, and didn't check the instances section.  My initial interpretation of "instances" section was the more instance specific parameter definitions within the common-instanceTemplate section.  Now I see what you are getting at.   Let me try another test..... ... View more