Thank you for the clarification.      D. ... View more

Thank you for the response.   To your point about bootstrapping/terminating - I have noticed during testing when I'm cycling through many bootstrapping / terminations that Director can get into a state where it seems to no longer respond to bootstrapping requests (from command line).    I'll typically let Director "cool down" for a few minutes and retry.   ... View more

Cleaned up the redundant common-instanceTemplate section per the latest spec.  Looks like Director is now accepting my custom image - however now I'm running into the other limitation that's been a thorn in my side, which is the fact that the custom image can't have data disks defined like they can in AWS.   I believe the workaround for that is to define disk count as 0, but I'll need to rebuild my worker packer image to include the additional data disks for hdfs.  Anyway, different subject.   Thank you Bill for the guidance.  I'll mark this issue as resolved.  ... View more

Bill, I belived I misinterpreted your initial suggestion.  I was focused on the common-instanceTemplate section entirely, and didn't check the instances section.  My initial interpretation of "instances" section was the more instance specific parameter definitions within the common-instanceTemplate section.  Now I see what you are getting at.   Let me try another test..... ... View more

Also tried updating hocon file to reflect the new formatting approach you referenced:   common-instanceTemplate { # Core config fields that are common to all node types base { type: STANDARD_DS13_V2 image: /subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55 useCustomManagedImage: Yes customImagePlan: "" ... ... ... } } # Core config fields that are common to all master nodes master-base: ${common-instanceTemplate.base} { computeResourceGroup: ${customer-params.resourceGroup} availabilitySet: ${customer-params.masterAvailabilitySet} storageAccountType: "Premium_LRS" dataDiskSize: 512 managedDisks: Yes publicIP: Yes } # Config fields for master-1 nodes master-1: ${common-instanceTemplate.master-base} { dataDiskCount: 1 instanceNamePrefix: master-1 publicIP: ${?common-instanceTemplate.master-base.publicIP} } ... ... ... Hitting same exception.  Including full trace...   ERROR [qtp187120217-21] 14c8453a-4e63-4fec-8eef-e7b587b0af52 POST /api/d6.2/environments/azuresb/deployments - - com.cloudera.launchpad.api.d6_0.response.ExceptionTranslatorD6_0: Found validation issues com.cloudera.launchpad.api.common.DeploymentsResource$InvalidDeploymentTemplateException: Validation errors: ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Image '/subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55' is referencing a Custom Image but not all required fields are set. To use Custom Images set the 'useCustomManagedImage' and 'customImagePlan' fields.}, causes=[]}Conditions: ValidationExceptionCondition{scope='azure.com.cloudera.director.azure.compute.provider.AzureComputeProvider.template', key='Optional.of(image)', exceptionInfo='ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Image '/subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55' is referencing a Custom Image but not all required fields are set. To use Custom Images set the 'useCustomManagedImage' and 'customImagePlan' fields.}, causes=[]}'} at com.cloudera.launchpad.api.common.DeploymentsResource.validate(DeploymentsResource.java:835) at com.cloudera.launchpad.api.common.DeploymentsResource.lambda$validateWithMarketplaceMode$0(DeploymentsResource.java:855) at com.cloudera.launchpad.pluggable.util.ConfigurationPropertiesUtil.validateWithMarketplaceMode(ConfigurationPropertiesUtil.java:106) at com.cloudera.launchpad.api.common.DeploymentsResource.validateWithMarketplaceMode(DeploymentsResource.java:850) at com.cloudera.launchpad.api.common.DeploymentsResource.create(DeploymentsResource.java:270) at com.cloudera.launchpad.api.common.DeploymentsResource$$FastClassBySpringCGLIB$$d7365ebd.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88) at com.cloudera.launchpad.api.common.ResourceMDCEnhancer.enhanceMDCForDeploymentApiCall(ResourceMDCEnhancer.java:179) at sun.reflect.GeneratedMethodAccessor458.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at com.cloudera.launchpad.api.common.DeploymentsResource$$EnhancerBySpringCGLIB$$5f181472.create(<generated>) at com.cloudera.launchpad.api.d6_1.DeploymentsResourceD6_1.create(DeploymentsResourceD6_1.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:215) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at com.cloudera.launchpad.DenyTraceTrackRequestFilter.doFilter(DenyTraceTrackRequestFilter.java:32) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at net.rakugakibox.spring.boot.logback.access.LogbackAccessSecurityAttributesSaveFilter.doFilter(LogbackAccessSecurityAttributesSaveFilter.java:28) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:155) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748)         ... View more

Still failing.   Here is what I tried...   common-instanceTemplate { # Core config fields that are common to all node types base { type: STANDARD_DS13_V2 #image: cloudera-centos-75-latest image: /subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55 useCustomManagedImage: Yes customImagePlan: "" ... ... ... master-base { computeResourceGroup: ${customer-params.resourceGroup} availabilitySet: ${customer-params.masterAvailabilitySet} storageAccountType: "Premium_LRS" dataDiskSize: 512 managedDisks: Yes useCustomManagedImage: ${?common-instanceTemplate.base.useCustomManagedImage} customImagePlan: ${?common-instanceTemplate.base.customImagePlan} publicIP: Yes ... ... ... # Config fields for master-1 nodes master-1 { dataDiskCount: 1 instanceNamePrefix: master-1 useCustomManagedImage: ${?common-instanceTemplate.base.useCustomManagedImage} customImagePlan: ${?common-instanceTemplate.base.customImagePlan} publicIP: ${?common-instanceTemplate.master-base.publicIP} }   I repeated this for each master instance, worker base, worker instance, and manager sections.   Director error:   ValidationExceptionCondition{scope='azure.com.cloudera.director.azure.compute.provider.AzureComputeProvider.template', key='Optional.of(image)', exceptionInfo='ErrorInfo{code=PROVIDER_EXCEPTION, properties={message=Image '/subscriptions/f5300b4d-bebc-4d95-8b6c-4ea77d9b3423/resourceGroups/Packer/providers/Microsoft.Compute/images/cloudera-packerbake-36.0.55' is referencing a Custom Image but not all required fields are set. To use Custom Images set the 'useCustomManagedImage' and 'customImagePlan' fields.}, causes=[]}'} ... View more

Thank you for the reply Bill.  You have confirmed my hunch.  I'm giving this a try now.  Will let you know. ... View more