Trying to stop cluster through cloudbreak but ran into a issue. The Ambari server is shown as unhealthy in cloudbreak ui but the server has no issues. I am able to ambari ui and stop and start the services from there just fine. Below is the stack trace found in docker cloudbreak logs.     2019-11-21 11:19:24,589 [reactorDispatcher-26] doExecute:58 WARN c.s.c.c.f.c.s.ClusterSyncActions - [owner:8bd40312-efc1-48b6-b4ef-7cee8ad660fd] [type:STACKVIEW] [id:1] [name:nifi-dev-cluster] [flow:bf84da9f-0ace-4ae8-a585-253db935b5d1] [tracking:39f41bf6-8d17-4d63-84c0-4489ec3bd43e] Error during executing cluster sync. java.lang.NullPointerException: Cannot get property 'items' on null object at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:60) at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:190) at org.codehaus.groovy.runtime.callsite.NullCallSite.getProperty(NullCallSite.java:46) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:298) at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper.getHostStatuses(ServiceAndHostService.groovy:93) at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper$getHostStatuses.call(Unknown Source) at com.sequenceiq.ambari.client.AmbariClient.getHostStatuses(AmbariClient.groovy) at com.sequenceiq.cloudbreak.service.cluster.ClusterService.updateClusterMetadata(ClusterService.java:682) at com.sequenceiq.cloudbreak.service.cluster.flow.status.AmbariClusterStatusUpdater.updateClusterStatus(AmbariClusterStatusUpdater.java:57) at com.sequenceiq.cloudbreak.reactor.ClusterSyncHandler.accept(ClusterSyncHandler.java:55) at com.sequenceiq.cloudbreak.reactor.ClusterSyncHandler.accept(ClusterSyncHandler.java:22) at com.sequenceiq.cloudbreak.reactor.ClusterSyncHandler$$FastClassBySpringCGLIB$$ab048b8d.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:56) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at com.sequenceiq.cloudbreak.reactor.ClusterSyncHandler$$EnhancerBySpringCGLIB$$e8ebcadb.accept(<generated>) at reactor.bus.EventBus$3.accept(EventBus.java:317) at reactor.bus.EventBus$3.accept(EventBus.java:310) at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72) at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51) at reactor.bus.EventBus.accept(EventBus.java:591) at reactor.bus.EventBus.accept(EventBus.java:63) at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160) at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2019-11-21 11:19:24,615 [reactorDispatcher-26] sendEvent:130 INFO c.s.c.c.f.AbstractAction - [owner:8bd40312-efc1-48b6-b4ef-7cee8ad660fd] [type:STACKVIEW] [id:1] [name:nifi-dev-cluster] [flow:bf84da9f-0ace-4ae8-a585-253db935b5d1] [tracking:39f41bf6-8d17-4d63-84c0-4489ec3bd43e] Triggering event: CLUSTERSYNCFAILHANDLEDEVENT, payload: com.sequenceiq.cloudbreak.reactor.api.event.StackEvent@7b0250a4   ... View more

Trying to create a HDF cluster in AWS using cloudbreak but the process is stuck in "Create in Progress". AWS console show the servers are created and they up and running. Was able to ssh from cloudbreak VM so there are connectivity issues. Cloudbreak logs shows no errors. These messages repeat over and over again. Checked the Ambari server and nginx was up and running. 2019-06-24 16:50:02,992 [reactorDispatcher-85] pollWithTimeout:32 INFO  c.s.c.s.PollingService - [owner:e315477d-074c-4ed1-ad47-1367a19b0567] [type:STACK] [id:10] [name:data-ingestion-hdf-dev] [flow:21049f09-4326-4668-b391-8972b1dfeccd] [tracking:40e8f3bf-54ad-4cd0-9307-ca4179bd2886] Polling attempt 3. 2019-06-24 16:50:02,993 [reactorDispatcher-85] checkStatus:25 INFO  c.s.c.s.s.f.NginxCertListenerTask - [owner:e315477d-074c-4ed1-ad47-1367a19b0567] [type:STACK] [id:10] [name:data-ingestion-hdf-dev] [flow:21049f09-4326-4668-b391-8972b1dfeccd] [tracking:40e8f3bf-54ad-4cd0-9307-ca4179bd2886] Check if nginx is running on 130.x.xx.xx:9443. 2019-06-24 16:50:12,274 [http-nio-8080-exec-8] <init>:57 INFO  c.s.c.f.MDCContextFilter - [owner:spring] [type:springLog] [id:] [name:] [flow:] [tracking:] No trackingId in request. Adding trackingId: '1d7362b4-ad15-46ee-8c6e-06ccb4ad86d4' 2019-06-24 16:50:12,276 [http-nio-8080-exec-8] getAllForAutoscale:311 INFO  c.s.c.s.StackCommonService - [owner:undefined] [type:Autoscale] [id:] [name:] [flow:] [tracking:1d7362b4-ad15-46ee-8c6e-06ccb4ad86d4] Get all stack, autoscale authorized only. 2019-06-24 16:50:14,357 [scheduledExecutor-1] distributeFlows:166 INFO  c.s.c.s.h.HeartbeatService - [owner:spring] [type:springLog] [id:] [name:] [flow:] [tracking:] Active CB nodes: (1)[[CloudbreakNode{uuid='52c98d7e-a196-4ec1-8378-5b8adfc2f83a', lastUpdated=1561395000002}]], failed CB nodes: (0)[[]] 2019-06-24 16:50:22,273 [http-nio-8080-exec-13] <init>:57 INFO  c.s.c.f.MDCContextFilter - [owner:spring] [type:springLog] [id:] [name:] [flow:] [tracking:] No trackingId in request. Adding trackingId: '71f18ec4-a5e4-4307-8fca-e6766fd8f6e6' 2019-06-24 16:50:22,275 [http-nio-8080-exec-13] getAllForAutoscale:311 INFO  c.s.c.s.StackCommonService - [owner:undefined] [type:Autoscale] [id:] [name:] [flow:] [tracking:71f18ec4-a5e4-4307-8fca-e6766fd8f6e6] Get all stack, autoscale authorized only. ... View more

Trying to create a HDF cluster using cloudbreak but the creation process is stuck at step Infrastructure metadata collection finished. When checking the cloudbreak logs saw the below: 2019-06-21 15:34:36,443 [reactorDispatcher-57] pollWithTimeout:32 INFO c.s.c.s.PollingService - [owner:e315477d-074c-4ed1-ad47-1367a19b0567] [type:STACK] [id:5] [name:dv-data-ingestion-hdf-dev] [flow:7ed0ed96-2d9b-47c5-8f3d-fa8fd91a5049] [tracking:8454d361-1d3c-4e7b-9b03-c9184450d47a] Polling attempt 35. 2019-06-21 15:34:36,445 [reactorDispatcher-57] checkStatus:25 INFO c.s.c.s.s.f.NginxCertListenerTask - [owner:e315477d-074c-4ed1-ad47-1367a19b0567] [type:STACK] [id:5] [name:dv-data-ingestion-hdf-dev] [flow:7ed0ed96-2d9b-47c5-8f3d-fa8fd91a5049] [tracking:8454d361-1d3c-4e7b-9b03-c9184450d47a] Check if nginx is running on 130.6.234.168:9443. I checked and the cloudbreak VM is able to connect to the Ambari server and nginx is running on the server. Nginx error logs shows this: 2019/06/21 15:33:16 [error] 4913#0: *16 connect() failed (111: Connection refused) while connecting to upstream, client: 135.28.27.13, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8080/favicon.ico", host: "130.6.234.168", referrer: "https://130.6.234.168/ambari/" ... View more