Any suggestions for why director is deploying conflicting versions of the agent and parcels?  Seems like a fundamental issue.  ... View more

We use templates to deploy our clusters.  I've discovered why the parcel activation is failing.  Version 6.2 agents are being installed while version 6.3 parcels are being distributed / activated.  This same template worked fine when 6 was originally released.  In the template we have: cluster { products { CDH: 6 We are not defining a parcel URL, which I assume means we are using the default based on the version of director, which is 6.3.0.  I've tried defining more specific version of CDH above (6.2, 6.3) but director continues to deploy 6.2 agents and 6.3 parcels. ... View more

I tried another deployment.  Here are some entires that standout in the manager server log - at least to me:   2019-08-15 00:09:29,753 WARN scm-web-93:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,754 INFO scm-web-93:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.158 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,755 WARN scm-web-177:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,757 INFO scm-web-177:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.246 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,771 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,774 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.244 over PT0.003S. Rate: 25.352 MB/s 2019-08-15 00:09:29,785 WARN scm-web-161:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,785 INFO scm-web-161:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.191 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,819 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,819 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.215 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,827 WARN scm-web-93:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,827 INFO scm-web-93:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.226 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,868 WARN scm-web-91:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,869 INFO scm-web-91:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.208 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,874 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,874 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.250 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,874 WARN scm-web-290:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,875 INFO scm-web-290:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.241 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,905 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,905 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.243 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,914 WARN scm-web-170:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,915 INFO scm-web-170:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.137 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,920 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,925 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.185 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,928 WARN scm-web-177:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,929 INFO scm-web-177:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.138 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:29,993 WARN scm-web-282:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,993 INFO scm-web-282:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.165 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:29,993 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:29,994 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.149 over PT0.001S. Rate: 76.056 MB/s 2019-08-15 00:09:30,018 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:30,018 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.150 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:30,030 WARN scm-web-94:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:30,030 INFO scm-web-94:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.188 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:30,034 INFO scm-web-89:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 172.24.1.165 2019-08-15 00:09:30,048 WARN scm-web-157:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent 2019-08-15 00:09:30,048 INFO scm-web-157:com.cloudera.server.web.cmf.ParcelController: Served parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent (79750 bytes) to 172.24.71.247 over PT0S. Rate: Infinity MB/s 2019-08-15 00:09:30,065 WARN scm-web-94:com.cloudera.server.web.cmf.ParcelController: No hash for parcel CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.torrent   Then there are several INFO entries which seem to indicate parcels are being distributed / activated successfully until we hit the following error, after which Altus Director fails the deployment:     /clusters/ez2/parcels/products/CDH/versions/6.3.0-1.cdh6.3.0.p0.1279813/commands/activate 2019-08-15 00:12:19,738 INFO scm-web-290:com.cloudera.parcel.components.ParcelManagerImpl: Activating parcel CDH:6.3.0-1.cdh6.3.0.p0.1279813 on cluster ez2 2019-08-15 00:12:19,756 INFO scm-web-290:com.cloudera.enterprise.AbstractWrappedEntityManager: Rolling back transaction that wasn't marked for rollback-only. java.lang.Exception: Non-thrown exception for stack trace. at com.cloudera.enterprise.AbstractWrappedEntityManager.close(AbstractWrappedEntityManager.java:161) at com.cloudera.cmf.persist.CmfEntityManager.close(CmfEntityManager.java:367) at com.cloudera.server.web.cmf.AuthFilterEntityManager.close(AuthFilterEntityManager.java:191) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:209) at com.cloudera.api.dao.impl.ManagerDaoBase.access$100(ManagerDaoBase.java:82) at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:239) at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32) at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210) at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:131) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:169) at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:272) at com.sun.proxy.$Proxy205.activate(Unknown Source) at com.cloudera.api.v3.impl.ParcelResourceImpl.activateCommand(ParcelResourceImpl.java:70) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:189) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:261) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:117) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99) at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:263) at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:189) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:299) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:218) at javax.servlet.http.HttpServlet.service(HttpServlet.java:665) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:274) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:200) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at com.cloudera.api.ApiBasicAuthFilter.doFilter(ApiBasicAuthFilter.java:86) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at com.cloudera.server.common.BoundedQueuedThreadPool$2.run(BoundedQueuedThreadPool.java:94) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) 2019-08-15 00:12:19,759 INFO scm-web-290:com.cloudera.api.ApiExceptionMapper: Exception caught in API invocation. Msg:The version change by activating the parcel would cause an error java.lang.IllegalArgumentException: The version change by activating the parcel would cause an error   ... View more

The log entry from Altus Director I provided mirrors the error I saw on the manager node while tailing the logs there. Parcel distribution / activation seemed to be going fine until it hit the error I posted. Unfortunately I've terminated the deployment as this was our 4th attempt. I was thinking of pinning CDH version in the cluster bootstrap to 6.2 instead of 6 so that it doesn't grab the latest (6.3) parcels. If I hit the same issue, I will rerun my original config and provide more logs from the manager node. ... View more

Thank you. Exactly what I was looking for. ... View more

How do I navigate to all my posts in the new layout? ... View more

I'm in the middle of another deployment where I see random failures reported by director when attempting to connect to port 22 of cluster instances during bootstrap.  I'm seeing this for deployments where I haven't enabled publicip - so it's no just for publicip enabled deployments.  Each time, it's only 1 or 2 of the instances and each time I can connect to the instance from the director server just fine, even during the retry window when director still claims it cannot connect.  Not sure why director thinks it can't connect when I clearly can from the same server.  I've confirmed the forward/reverse DNS resolution is working.  I basically have to deploy several times before I hit the sweet spot and the deployment completes.  This type of inconsistent behavior has been systemic with Azure deployments.  I've experienced none of this with AWS deployments.  I'll attempt to collect more logs. ... View more

Thanks for your reply.   Bootstrap did fail in this case, which surprised me given the fact that during the retry window I could ssh to the instance's private IP from Director.  Also, Director had no problem connecting to the other 2 master nodes that I enabled public IP on.  The log entries I provided are repeated over and over until the retries are exhausted.   ... View more