Created 12-23-2016 03:31 PM
Unable to add a datanode in cloudbreak , cloudbreak has following log entries
/cbreak_cloudbreak_1 | 2016-12-15 03:36:41,762 [containerBootstrapBuilderExecutor-1] call:71 WARN c.s.c.o.OrchestratorBootstrapRunner - [owner:b7f4fd63-4e2d-4fd5-91c9-45faa7f6ac7a] [type:STACK] [id:10] [name:abcd] Orchestrator component Salt failed to start, retrying [1/90] Elapsed time: 2067 ms, Total elapsed time: 2067 ms, Reason: There are missing nodes from salt network response: [Node{privateIp='10.92.1.13', publicIp=‘W.X.Y.Z', hostname='null', hostGroup='host_group_slave_1', dataVolumes=null}], additional info: SaltBootstrap{gatewayConfig=GatewayConfig{publicAddress=‘W.X.Y.Z', privateAddress='10.92.1.12', certificateDir='/certs/stack-10', serverCert='/certs/stack-10/ca.pem', clientCert='/certs/stack-10/cert.pem', clientKey='/certs/stack-10/key.pem', saltPassword=''}, originalTargets=[Node{privateIp='10.92.1.13', publicIp=‘W.X.Y.Z', hostname='null', hostGroup='host_group_slave_1', dataVolumes=null}], targets=[Node{privateIp='10.92.1.13', publicIp=‘W.X.Y.Z', hostname='null', hostGroup='host_group_slave_1', dataVolumes=null}]} /cbreak_cloudbreak_1 | 2016-12-15 03:36:51,764 [containerBootstrapBuilderExecutor-1] isExitNeeded:91 DEBUG c.s.c.o.OrchestratorBootstrapRunner - [owner:b7f4fd63-4e2d-4fd5-91c9-45faa7f6ac7a] [type:STACK] [id:10] [name:abcd] exitCriteriaModel: ClusterDeletionBasedExitCriteriaModel{stackId=10, clusterId=null}, exitCriteria: com.sequenceiq.cloudbreak.core.bootstrap.service.ClusterDeletionBasedExitCriteria@1295d005 /cbreak_cloudbreak_1 | 2016-12-15 04:16:00,630 [reactorDispatcher-65] createStackEvent:71 INFO c.s.c.s.e.DefaultCloudbreakEventService - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Created stack event: CloudbreakEvent{id=576, eventType='UPDATE_FAILED', eventTimestamp=Thu Dec 15 04:16:00 UTC 2016, eventMessage='New node(s) could not be added to the cluster. Reason com.sequenceiq.cloudbreak.core.CloudbreakException: com.sequenceiq.cloudbreak.orchestrator.exception.CloudbreakOrchestratorFailedException: java.util.concurrent.ExecutionException: com.sequenceiq.cloudbreak.orchestrator.exception.CloudbreakOrchestratorFailedException: There are missing nodes from job result: [10.92.1.13]', owner='b7f4fd63-4e2d-4fd5-91c9-45faa7f6ac7a', account='seq1234567', cloud='AZURE_RM', region='Canada Central', blueprintName='main-hdp-spark-cluster-ifc-hadoop', instanceGroup='null', blueprintId=6', stackId=10', clusterId=10', clusterName=ABCD}/cbreak_cloudbreak_1 | 2016-12-15 04:16:00,631 [reactorDispatcher-65] accept:38 INFO c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Sending notification with data: CloudbreakEvent{id=576, eventType='UPDATE_FAILED', eventTimestamp=Thu Dec 15 04:16:00 UTC 2016, eventMessage='New node(s) could not be added to the cluster. Reason com.sequenceiq.cloudbreak.core.CloudbreakException: com.sequenceiq.cloudbreak.orchestrator.exception.CloudbreakOrchestratorFailedException: java.util.concurrent.ExecutionException: com.sequenceiq.cloudbreak.orchestrator.exception.CloudbreakOrchestratorFailedException: There are missing nodes from job result: [10.92.1.13]', owner='b7f4fd63-4e2d-4fd5-91c9-45faa7f6ac7a', account='', cloud='AZURE_RM', region='Canada Central', blueprintName='', instanceGroup='null', blueprintId=6', stackId=10', clusterId=10', clusterName=ABCD}
Created 12-23-2016 03:33 PM
Found an internal bug (https://hortonworks.jira.com/browse/BUG-58873) but didn't get much details there. @Attila Kanto
Created 12-23-2016 04:21 PM
Which version of Cloudbreak do you use?
Could you attach the full cloudbreak log here, please?
Can you tell me the steps you made? Did you try to upscale with one node on a slave or worker hostgroup?